http://www.nytimes.com/2014/02/09/us/snowden-used-low-cost-tool-to-best-nsa.html?pagewanted=all&_r=1
NYT says Snowden used a web crawler to snag info. They neglect to tell us which one. I'd be interested to know what software he used. (Could come in handy)
QuoteAgency officials insist that if Mr. Snowden had been working from N.S.A. headquarters at Fort Meade, Md., which was equipped with monitors designed to detect when a huge volume of data was being accessed and downloaded, he almost certainly would have been caught. But because he worked at an agency outpost that had not yet been upgraded with modern security measures, his copying of what the agency's newly appointed No. 2 officer, Rick Ledgett, recently called "the keys to the kingdom" raised few alarms.
Err... that's a bit backwards isn't it? So rather than having something server-side to flag up when someone is accessing loads of data, they instead rely in each system to report how much it is accessing.
To: America
Subject: Your secrets
Body: Assume you have none
Quote from: Rooftop on February 10, 2014, 09:06:40 AM
something server-side
If you have massive data sets separated across several servers, that might not be that simple. What's odd is that they took a blacklist approach instead of a whitelist approach.
I once worked in a secret environment. People were incredibly cavalier about it and most engineers regarded the security people as an obstacle to getting work done, demanding unreasonable things like that we put away secret papers and lock our file cabinets when we went to lunch. On any given day at lunch hour there were half-deserted cubicle farms with manila folders stamped SECRET sitting out in the open. A Russian spy (this was the 1980s) in a moving company uniform could have scooped up a file cabinet's worth and put the cabinet on a hand truck and left the building. The security were always threatening people with disciplinary action, but seemed to have no real authority and were ignored, not least by me (I did get them really mad once with a bit of sarcasm, but they didn't actually add anything to my file).
One guy was behind on his work and took a large set of secret papers home with him to work on - strictly verboten. It probably would have worked out okay for him, except he stopped for groceries on the way home and his car was stolen. Oops. I think he actually got fired over that.
wget -mirror FTW :)
wget is pretty capable, most people just use it to retrieve a single page and have no idea how well it works as a crawler. It does a good job of rewriting links to be locally accessible too.
I read somewhere it was 'more sophisticated' than the wget that Bradley Manning used.
I'm guessing curl.
Idiots.
The old ways are the best, trust-relationship exploitation, probably anything going to a vetted contractor wasn't transactoin-volume monitored.
>Idiots.
Yes :) It's entertaining but also scary.
Now I'm thinking about mimicked human behavior, random timers, throttlers, a few user_names and proxies and other things while I should be working ;)
I know nothing about security but perhaps trying to enforce everybody to use an actual browser login would a good extra security step.
For a 'poor man's' form spam trap I often add a few fields to my forms that are not visbile to humans with a few varying (CSS) tricks, and switch out the fields that should be submitted that day.
For example, one day the expected field for username might be 'user_name' , the next day that field is the trap and some other field will be the input for user_name.
Fill in the wrong one and you're a bot. Send in the suits!