Over the past few weeks, I’ve increasingly heard of spam and abuse problems originating in Amazon EC2.
This has culminated in a blog post yesterday by Brian Krebs at the Washington Post:
It took me by surprise this weekend to discover that that mounds of porn spam and junk e-mail laced with computer viruses are actively being blasted from digital real estate leased to [Amazon].
He goes on to discuss how EC2 space is now actively blocked by Outblaze, and has been listed by Spamhaus in their PBL list. A spokesperson for Amazon said:
“We have a clear acceptable use policy and whenever we have received a complaint of spam or malware coming through Amazon EC2, we have moved swiftly to strictly enforce the use policy by network isolating (or even terminating) any offending instances,” Kinton said. She added that Amazon has since taken action against the EC2 systems hosting the [malware].
However as Seth Breidbart noted in the comments, ‘note that Amazon will terminate the instance. That means that the spammer just creates another instance, which gets a new IP address, and continues spamming.’ True enough – as described, instance termination simply isn’t good enough.
My recommendations:
as John Levine noted, it’s likely that Amazon need to treat EC2-originated traffic similarly to how an ISP treats their DSL pools – filtering outbound traffic for nastiness, in particular rate-limiting port 25/tcp connections on a per-customer basis, so that an instance run by (or infiltrated by) a spammer cannot produce massive quantities of spam before it is detected and cut off.
However, I’m not talking about blocking port 25/tcp outbound entirely. That’s not appropriate — an EC2 instance is analogous to a leased colo box in a server farm, and not being able to send mail from our instances would really suck for EC2 users (like myself and my employers).
It would help if there were a way to look up customer IDs from the IP address of the EC2 nodes they’re using — either via WHOIS or through rDNS. Even an opaque customer ID string would allow anti-abuse teams to correlate a single customer’s activity as they cycle through EC2 instances. This would allow those teams to deal with the reputation of Amazon’s customers, instead of Amazon’s own rep, analogous to how “traditional” hosters use SWIP to publicize their reassignments of IPs between their customers.
There’s some more discussion buried in a load of knee-jerking on the NANOG thread. Here’s a few good snippets:
Jon Lewis: ‘I got the impression the only thing Amazon considers abuse is use of their servers and not paying the bill. If you’re a paying customer, you can do whatever you like.’ (ouch.)
Ken Simpson: ‘IMHO, Amazon will eventually be forced to bifurcate their EC2 IP space into a section that is for “newbies” and a section for established customers. The newbie space will be widely black-listed, but will also have a lower rate of abuse complaint enforcement. The only scalable way to deal with a system like EC2 is to provide clear demarcations of where the crap is likely to originate from.’
Bill Herrin: ‘From an address-reputation perspective EC2 is no different than, say, China. Connections from China start life much closer to my filtering threshold that connections from Europe because a far lower percentage of the connections from China are legitimate. EC2 will get the same treatment.’
There’s also an earlier thread here.
Anyway, this issue is on fire — Amazon need to get the finger out and deal with it quickly and effectively, before EC2 does start to run into widespread blocks. I’m already planning migration of our mail-sending components off of EC2; we’re already seeing blocks of mail sent from it, and it’s looking likely that these will increase. :(
(It’s worth noting that a block of EC2’s netblocks today will produce a load of false positives, mainly on transactional mail, if you’re contemplating it. So I wouldn’t recommend it. But a lot of sites are willing to accept a few FPs, it seems.)
Tags: abuse, amazon, anti-spam, aws, blocklists, ec2, malware, policy
Twitter has this nasty habit — if you come across a tweet in your feed reader containing a URL, and you want to follow that link, you can’t, because Twitter doesn’t auto-link URLs in its RSS feeds. Instead, you have to click on the feed item, itself, wait for that to open in the browser, then click on the link in the new browser tab. That link will, in turn, open in another new tab.
Here’s a quick-hack Greasemonkey user script to inhibit this second new-tab:
Tags: greasemonkey, hacks, software, twitter, userscript, web
‘You’ll know what my riddle means
When you’ve eaten mangosteens.’
– The Crab That Played with the Sea, by Rudyard Kipling
When I travelled through Thailand, I got rightly hooked on the delicious mangosteen, traditionally dubbed the “Queen of Fruit” by the Thais. I’ve been keeping an eye out ever since, through our travels to the US and back, without any luck. (In particular, they’ve been blocked by US customs for a long time, although reportedly this is changing nowadays.)
Finally, last year, they appeared in our local Tesco supermarket here in Ireland — or at least, an empty box appeared, sans fruit! That was it, though, until a couple of weeks ago, when my friend Bob was lucky enough to come across a few, and grabbed 4 for me. (Thanks Bob!)
It appears they’re in season around the start of June, which is when they make it to Tesco’s. Naturally, they’re much more expensive here — Tesco were selling them for about EUR 1.20 each, whereas a bag of 30 were about 50 cents when we used to buy them at the street-side in Ko Chang. But that’s to be expected, really.
Since they’re tricky enough to get hold of, I thought I should document exactly what to do with them once you get ‘em ;)
They start off looking like this, roughly tomato-sized fruit with a thick, papery rind:
Get your thumbnail into the rind, not too deep though!, and tear it off like so:
Look at the rind’s great colour! Watch out for it, though, as it stains clothing easily. Discard the rind, and pluck out the fleshy, juicy white segments:
(Pay no attention to their resemblance to testicles. ;)
Finally you’ll wind up with 6 or so seedless segments, and 1 or 2 seed-bearing segments, larger than the others, containing a large inedible seed along with a fair bit of flesh:
Eat ‘em and enjoy the flavour — it’s a bit like a tart, vanilla-y peach, but juicier, creamier and much smoother in texture. Mmmm, truly delicious. I’m looking forward to picking up some more soon!
I considered planting the seeds, but unfortunately, you can forget about growing a tree in your back yard; the mangosteen tree requires a tropical climate:
‘The mangosteen is ultra-tropical. It cannot tolerate temperatures below 40º F (4.44º C), nor above 100º F (37.78º C). Nursery seedlings are killed at 45º F (7.22º C).’
Ah well. Seems I’ll be at Tesco’s mercy for more.
Joey Hess suggests that current discussions about the superfluity of DVCS systems have a parallel in how the internet protocol world, circa 1993, played out:
I’m reminded of 1993. Using the internet at that time involved using a mishmash of stuff — Telnet, FTP, Gopher, strange things called Archie and Veronica. Or maybe this CERN “web” thing that Tim Berners-Lee had just invented a few years before, but that mostly was useful to particle physicists.
Then in 1994 a few more people put up web sites, then more and more, and suddenly there was an inflection point. Suddenly we were all browsing the web and all that other stuff seemed much more specialised and marginalised.
I would disagree, a little. Back in the early ’90’s, I was a sysadmin playing around with internet- and intranet-facing TCP/IP services (although in those days, the term “intranet” hadn’t been coined yet), so I gained a fair bit of experience at the coal-face in this regard. The mish-mash of protocols – telnet, gopher, Archie, WAIS, FTP, NNTP, and so on — all had their own worlds and their own views of the ‘net. What changed this in 1993 was not so much the arrival of HTTP, but TimBL’s other creation: the URL.
The URL allowed all those balkanized protocols to be supported by one WWW client, and allowed a HTML document to “link” to any other protocol –
The WWW browsers can access many existing data systems via existing protocols (FTP, NNTP) or via HTTP and a gateway. In this way, the critical mass of data is quickly exceeded, and the increasing use of the system by readers and information suppliers encourage each other.
This was a great “embrace and extend” manoeuvre by TimBL, in my opinion — by embracing the existing base of TCP/IP protocols, the WWW client became the ideal user interface to all of them. Once NCSA Mosaic came along, there really was no alternative to rival the Web’s ease of use. This was the case even if you didn’t have a HTTP server of your own; you could still access HTML documents and remote URLs.
In essence, HTML and the URL were the trojan horse, paving the way for HTTP (as HTML’s native distribution protocol) to succeed. It wasn’t the web sites that helped the WWW “win”, but embrace-and-extend via the URL.
For what it’s worth, I think there is an interesting parallel in today’s DCVS world: git-svn.
Tags: dvcs, git, history, html, http, internet, protocols, web, www
Happy Firefox Download Day — or rather, Firefox Download Evening!
It turns out that the “day” in question has been defined as a 24-hour period starting at 10am Pacific Time; rather than compensating for the effects of timezones around the world, they’ve just picked an arbitrary 24-hour period.
That’s 6pm in Irish time, for example. At least I’m not one of the 57,000 Japanese pledgers, who’d be waiting up until 2am to kick off their download. It seems a little bizarre that there’s little leeway provided for non-US downloaders, who are right now twiddling their thumbs, waiting, while their “day” passes.
Annoyingly, the main world record page simply says ‘the official date for the launch of Firefox 3 is June 17, 2008′ — no mention of a starting time or official timezone at all!
This is the top thread on their forum right now — in addition to the omission of an entire continent ;)
Tags: downloads, firefox, international, open-source, time, timezones, us
On programmers “going dark” — Aristotle Pagaltzis writes:
Jeff Atwood argues that open source projects are in real danger of programmers “going dark,” which means they lock themselves away silently for a long time, then surface with a huge patch that implements a complex feature.
It seems to me that this is as much a technological problem as a social issue… and that we have the technological solution figured out: it’s called distributed version control. It means that that lone developer who locked himself in a room need not resurface with a single huge patch – instead, he can come back with a branch implementing the feature in individually comprehensible steps. At the same time, it allows the lone programmer to experiment in private and throw away the most embarrassing mistakes, addressing part of the social problem.
However, I don’t think he realised that the Jeff Atwood story he responded to was in fact an echo of Ben Collins-Sussman’s original article, where he specifically picked out DVCS as a source of this danger:
A friend of mine works on several projects that use git or mercurial. He gave me this story recently. Basically, he was working with two groups on a project. One group published changes frequently…
“…and as a result, I was able to review consistently throughout the semester, offering design tweaks and code reviews regularly. And as a result of that, [their work] is now in the mainline, and mostly functional. The other group [...] I haven’t heard a peep out of for 5 months. Despite many emails and IRC conversations inviting them to discuss their design and publish changes regularly, there is not a single line of code anywhere that I can see it. [...] Last weekend, one of them walked up to me with a bug [...] and I finally got to see the code to help them debug. I failed, because there are about 5000 lines of crappy code, and just reading through a single file I pointed out two or three major design flaws and a dozen wonky implementation issues. I had admonished them many times during these 5 months to publish their changes, so that we (the others) could take a look and offer feedback… but each time met with stony silence. I don’t know if they were afraid to publish it, or just don’t care. But either way, given the code I’ve seen, the net result is 5 wasted months.”
Before you scream; yes yes, I know that the potential for cave-hiding and writing code bombs is also possible with a centralized version control system like Subversion. But my friend has an interesting point:
“I think this failure is at least partially due to the fact that [DVCS] makes it so damn easy to wall yourself into a cave. Had we been using svn, I think the barrier to caving would have been too high, and I’d have seen the code.”
In other words, yes, this was fundamentally a social problem. A team was embarrassed to share code. But because they were using distributed version control, it gave them a sense of false security. “See, we’re committing changes to our repository every day… making progress!” If they had been using Subversion, it’s much less likely they would have sat on a 5000 line patch in their working copy for 5 months; they would have had to share the work much earlier.
To be honest, I’d tend to agree with Aristotle; just because centralized VC makes it harder to maintain a “private branch” with this “high barrier to caving”, and this therefore imposes a technical pressure to fix a social problem, doesn’t mean that is a good thing. I’d prefer to fix the DVCS to apply social pressure, and have both working tools and a working social organisation.
Another commenter on Ben’s original post put it well:
I [..] disagree, strongly, that DVCS makes code hiding any more difficult than single-branch VCS. When using a single branch, it’s usually a very small group of people who are allowed to commit. Any patches from non-core contributors get lost in a tangle of IRC pastebins, mailing lists, bug trackers, and blog posts. Furthermore, even if these patches are eventually committed, they have lost all their associated version information — the destructive rebase you complain about. DVCS allows anybody to branch from trunk, record their changes, and publish their branch in a service like Launchpad or github. For an example of this, look at the mass of user-created branches for popular projects like GNOME Do or AWN.
It’s very interesting to see those Launchpad sites, in my opinion.
I’ve spent many years shepherding contributions to SpamAssassin through our Bugzilla. We’ve often lost rule contributors, who are particularly hard to attract for some reason, due to delays and human overhead involved in this method. :( So an improved interface for this would be very useful…
Tags: coding, dvcs, git, mercurial, software, svn, version-control
So, Nelson is apparently contemplating a trip to Ireland, and was looking for tips. Since he’s not the first to ask, I thought I’d do some research among my friends on things to do and good places to stay and eat in our native country. Here’s the result.
First off — it’s worth noting that we’re all thirty-somethings, so backpacker stuff and heavy boozing is no longer on the menu. If you’re after that, though, head for Temple Bar in Dublin ;) This is mainly nice hotels, good food, and interesting things to look at.
To start with, I’d recommend driving as a means of getting around. Lots of the good stuff can’t be reached any other way, and the roads are generally pretty good nowadays (if a little narrow).
Prepare for rain.
Things to do: Connemara and Kerry are stunning; in my opinion, they’re unmissable, if you’re coming to Ireland in search of natural beauty. Clare and West Cork are pretty good too. Generally, the west coast is the place to go.
A friend recommends the Skelligs: ‘the best thing I’ve seen in Ireland. If its sunny. If its raining it sucks so don’t go.’ (I’ve never been — appalling, given that my great-grandfather wrote one of the definitive works on them, I need to fix that.)
Stuff to avoid: Dublin’s not too hot, unfortunately. Over-priced and hard to get around due to traffic. I mean, it’s quite nice, especially to live in, but as a tourist destination compared to other cities around the world I don’t quite get the attractiveness. Also, the south-east corner of the country, while full of nice friendly people, is exorbitantly expensive in my experience (even pricier than Dublin!), short on good stuff to see, and a bit of a washout, so I say skip it. (I have no idea why it’s so expensive, BTW. my theory is that it’s a traditional in-country holiday venue for Dubliners, and the Wexford inhabitants love to fleece us, so we got fleeced. whatever.)
In general, I’d say the larger towns aren’t too exciting; stick to the country.
The Lonely Planet guide to Ireland, while frequently backpacker-oriented, is pretty good for non-backpacker stuff as well. If you’re driving around, it’s a good source of offbeat stuff to check out. I used it a lot when driving around Connemara last year. They also do a great book of hikes which I can recommend.
Next, places to stay… that friend again: ‘if you’re doing the Ring of Kerry, I strongly recommend diverting to Valentia and staying in Glanleam House (beautiful grub, beautiful gardens, cheap) and doing a day trip from there to the Skelligs.’
Temple House in Sligo also comes recommended: ‘a classical Georgian mansion set in an estate of 1,000 acres, overlooking a 13th century lakeside castle of the Knights Templar.’
There are lots of useless hotel/B&B sites in Google, making it hard to tell crap from quality. But these sites come recommended:
Ireland’s Blue Book - ‘luxury accommodation in Irish Country House Hotels, Manor Houses and Castles. Also listed are Ireland’s finest gourmet restaurants.’ This is high-end stuff, but it’s pretty reliable, as far as I can see.
Friendly Homes of Ireland - another friend says ‘aka crazy houses of Ireland — terrible webpage, but good accommodation (its also a more attractive guide). We stayed here and loved it.’
Hidden Ireland - ‘a unique collection of historic private houses which provide the very best and most stylish country house accommodation available in Ireland - great Irish hospitality at an affordable price. Our houses are not hotels and are very much more than ordinary guesthouses. They all offer a rare opportunity to experience the lifestyle of a bygone age - a special and fascinating alternative to conventional tourist accommodation.’
Irish Landmark Trust, if you’re interested in self-catering stays at heritage houses.
Georgina Campbell guidebooks are apparently quite good.
Finally, scams and rip-offs are few and far between, so that’s not something to worry about. Crappy service and mediocre food, however, is more likely to be the source of problems. At least you can now get decent espresso pretty much everywhere!
Hope that helps someone ;) Got tips of your own? Feel free to add comments!
Tags: connemara, holidays, hotels, ireland, kerry, tourism, tourist, vacation
TypePad AntiSpam looks pretty cool. I’ve been trying it out for the past week on taint.org and underseacommunity.com, with no false positives or false negatives so far (although mind you I don’t get much spam, anyway, on those blogs, fortunately). Both are WordPress blogs — I set up Akismet, got a TypePad API key, and edited 3 lines in “wp-content/plugins/akismet/akismet.php”, and I was off.
However, here’s the key bit, the bit I’m most excited about — /svn/antispam/trunk/, particularly the GPL v2 LICENSE file — a fully open source backend!
The backend is a perl app built on Gearman and memcached. It uses DSpam instead of SpamAssassin, but hey, you can’t have everything ;) Nice, clean-looking perl code, too. Here’s hoping I get some tuits RSN to get this installed locally…
Tags: anti-spam, blog-spam, open-source, perl, typepad