The kiss you can remember
Is not a real kiss
Just as the circle you can see
Is not a true circle
The perfect kiss is as pleasant dream
Pure experience untainted
By awful reality of tangibles
That everywhere surrounds us
You must emerge from it
With but vague recall
Too fleeting to capture
Too slippery to grasp
Too pristine to bind
By fetters of finite memory
You will remember starting
But should you bethink anything after
Then, my friend, I'm afraid
You must try again
And again
Until mindlessness becomes you
Copyright (c) 2008, Anand Venkataraman I find it interesting that much bandwidth is spent describing what is a purely experiential thing. Indeed, words mean nothing here, and it is an irony of life that we often waste the most words in talking precisely about things over which words are so poignantly powerless!" ]]>
On a Linux command line, you want something like:
echo "SELECT MEDIAN(foo) FROM bar" | mysql -p mydb
echo "SELECT AVG(foo) FROM bar" | mysql -p mydb
The answer to this question is what potentially distinguishes a good engineer or scientist from a good manager. A good manager is focused on getting the job done, and does not care too much about exactly how it is done as long as she is confident it's being done efficiently. A good scientist or engineer is naturally curious, and wants to solve the problem a particular way just to know whether it can be done or not. The good engineer will most likely come up with one of the many excellent answers on the MySql forum. A good manager will likely say, "Is there a reason it has to be done in MySql? 'Cos I don't know the answer off the top of my head, but I'm sure I can find it soon enough if I browse a few minutes. If you just want the median efficiently, and don't care how I get it, I'd probably just use some combination of MySql and Perl/php/whatever:
echo SELECT foo FROM bar ORDER BY foo \
| mysql -N -p mydb \
| perl -e 'use POSIX qw(floor ceil);
@f = <>;
($a,$b) = (floor(@f/2-0.5), ceil(@f/2-0.5));
$median = ($f[$a]+$f[$b])/2;
print "$median\n";
'
echo "SELECT foo FROM bar order by foo" | mysql -p mydb -N | median
echo "SELECT foo FROM bar" | mysql -p mydb -N | median-unsorted
use DBI;
use POSIX qw(floor ceil);
...
my $sth = dbh->prepare('SELECT foo FROM bar ORDER BY foo");
$sth->execute();
my @f = $sth->fetchrow_array();
my($a,$b) = (floor(@f/2-0.5), ceil(@f/2-0.5));
my $median = ($f[$a]+$f[$b])/2;
...
SELECT AVG(foo) FROM (SELECT foo FROM bar ORDER BY foo limit m,n) as x
Finally, lest you worry about the memory requirements imposed by sucking in an entire column into a Perl or other such array let me reassure you that (a) Perl's implementation of in-memory arrays is very efficient, (b) you can easily read in arrays with millions of rows without taking up more than a few tens of megabytes of memory and (c) if, in fact, your table had many billions of rows, you should be able to get a very good approximation to your true median by sampling every nth row (Why?). This last point is especially relevant to scalable data analysis. You can calculate your answer as a median of medians by farming out computations using such paradigms as MapReduce. ]]>
]]> Now I'm not arguing that even our humane interrogation techniques are immoral and therefore should be abolished. Nor am I calling for greater cooperation from web-site owners in sharing sensitive information with the government. I'm only pointing out the poignant inconsistency in people's perceptions of what is allowable and what is not. Indeed, it is a rather paradoxical inconsistency, for on the one hand we sometimes allow the system to probe what we cherish as our most private and sacred sanctuaries, but on the other, we clamor with our loudest voices when the same system attempts to investigate information on the Internet, arguably many times removed from our mind-cores!
The righteousness (or lack of it) of various investigative procedures is not my concern here. But I am interested in exploring what persons who are genuinely concerned about the loss of their privacy might do to thwart authoritarian attempts to invade it. After all, do we not assume that it is natural for a suspect to attempt concealment despite being badgered at the stand? And do we not think it reasonable that polygraph test takers have at least tried to learn techniques to beat it? Why else would we find so much humor in George Costanza's parting advice to Seinfeld, who has to take the polygraph to prove he doesn't like Melrose place? "Remember," he says "It's not a lie if you believe it." That is is the perfect analogue to wiping your tracks clean on the Internet. A probe will discover nothing.
So having said that, we now ask: Is it possible to have an absolutely anonymous Internet experience - a veritable no-strings-attached one-night-stand with your browser? I confess it scares me a little to know that there are people who might want this - I mean, concern for privacy and a desire to enjoy an unsnooped life by making it "hard" for people to track you is one thing. But going out of one's way to make it practically impossible to associate one's online activity with oneself is borderline creepy. Who would want to do this?
The fact, however, is that most people are at the other end of the spectrum. They are naive enough to not know even to clear their browser histories and cookies when they leave their stations. Over the last day or so, we quickly put together a website using entirely free resources on an old PC (Clearly, with more resources, the scope for tracking and profiling users is vastly greater). The page shows how much information about users is likely logged by all the websites they visit. It's worth visiting at http://pippini.com
Having been in online advertising long enough and being aware as I am of the information that we unwittingly make available to humongously hungry hosts out there, I confess to still being mildly unsettled to see all of the information distilled and presented in one place like this. But yet, I typically settle for the most common precautions one should take, like clearing browser caches periodically, and refreshing DHCP addresses once in a while. If I wanted to be absolutely anonymous, the effort involved is simply too much and is not worth it for most of us. Nevertheless, here are three precautions such a person might be inclined to take, and actions authorities might assume to address attempts at attaining absolute anonymity.
]]>
$ gawk 'BEGIN {for(i=0;i<1e4;i++)print rand()+rand()+rand()}' \
| histo -stars -scale 50 -interval 0.15
# NumSamples = 10000; Max = 2.94838; Min = 0.0360424
# Mean = 1.5030500454; Variance = 0.253799739614845; SD = 0.503785410283828
# Each * represents a count of 50
0.0360 - 0.1860 [ 13]: *
0.1860 - 0.3360 [ 58]: **
0.3360 - 0.4860 [ 125]: ***
0.4860 - 0.6360 [ 250]: *****
0.6360 - 0.7860 [ 374]: ********
0.7860 - 0.9360 [ 552]: ************
0.9360 - 1.0860 [ 758]: ****************
1.0860 - 1.2360 [ 942]: *******************
1.2360 - 1.3860 [ 1026]: *********************
1.3860 - 1.5360 [ 1130]: ***********************
1.5360 - 1.6860 [ 1145]: ***********************
1.6860 - 1.8360 [ 1006]: *********************
1.8360 - 1.9860 [ 854]: ******************
1.9860 - 2.1360 [ 652]: **************
2.1360 - 2.2860 [ 501]: ***********
2.2860 - 2.4360 [ 306]: *******
2.4360 - 2.5860 [ 174]: ****
2.5860 - 2.7360 [ 108]: ***
2.7360 - 2.8860 [ 25]: *
2.8860 - 3.0360 [ 1]: *
... ]]>
First appeared in San Jose Mercury News Forums, Nov 18, 2006.
Consider the following ad, the likes of which we are all only too used to routing promptly into our spam buckets.
"The sex of your baby predicted CHEEP! Guaranteed Results. 150% REFUND for wrong predictions"
There you have it - a simple and sweet business plan if ever there was one. But people who ponder can promptly peer past its imposturous garb. This is just gambling in disguise. Suppose the service costs $1.00 and a thousand naive couples paid for it. On average, five hundred of them will receive a refund of $1.50; the rest will presumably be happy to have been told the sex of their baby. Despite the refunds, however, the service provider is $250 better off than she was before, and for no more effort than flipping a coin for each customer. Immediately, one can tell that the scheme is a rip-off, aimed at defrauding innocent parents-to-be of precious baby-dollars.
To the uninitiated, that is exactly how CPA advertising is being framed by ad-networks steeped in profiting from traditional methods: "Give us your ads. We'll show them. You only pay for ads that result in a sale, but not a cent on the rest." On the surface, yes; this does look like a scam. But fortunately, this is not what CPA is about. In fact, CPA represents the exact opposite. CPA ad-networks don't just flip a coin to show an ad. Considerable effort and investment go into determining where each ad is most likely to appeal, and, of course, web real-estate on which to show ads does not come free either. A CPA network will actually "lose" money if you don't make it and therefore it is in their best interest to make sure you are successful.
]]> I had always thought that CPA would be the fairest scheme of all to advertisers and that its basic principles would be straightforward enough to grasp immediately. But that seems not to be the case, at least as evidenced by a number of articles that describe misconceived notions of what CPA is all about. As recently as last week, Kevin Lee's column in ClickZ claimed "Selling inventory on a CPM guarantees a return ... CPA publishers are much happier piggybacking on demand you've already created than actually creating demand", a telling revelation that bespeaks signal ignorance about how CPA works.
Consider this: After many months of unsuccessfully trying to sell his house on his own, John finally decides to use the services of a professional. Now there's Mike, an established realtor who gladly agrees to stage John's home for several viewings but only in return for a fee for each staging, plus a percentage-based commission should the house ever get sold. But there's also Bob, a young and enthusiastic newbie, who is happy to stage John's home for free, and only takes the percentage-based sale commission. Which one of the two should John choose?
Here is a glimpse into John's train of thought. "Well, Mike's an established realtor. He has a number of contacts and is probably better equipped to get viewings from serious buyers. On the other hand, what does Mike care if my home doesn't sell? He's getting a staging fee anyway, and for all I know, that's probably what he makes his living from. But here's Bob, fresh out of ivy-league school. He claims to have done a lot of research and promises to stage my home in front of the few potential buyers that really matter. What do I have to lose by giving Bob a chance? Nothing really. Mike? About $200 a showing! So if I go with Mike, I'm just really paying to get my house in front of his elite-clients. Hmm..."
At this point John probably wants to find out if Mike's clients are exclusive to him or whether Bob also has access to them. Fortunately, soon he'll find out that there are no exclusive deals on the buyer side. Home buyers are a precious few, and they prefer not to get locked to specific realtors. With his latest hi-tech precision guided methods, Bob has most likely already identified and approached these very buyers to arrange a viewing of John's home. Clearly then, it makes more sense for John to go with Bob, or at the very least give Bob a chance while retaining Mike on the side. If Mike has an exclusivity-clause that forbids John from using the services of Bob, well then - there need be no more explicit a signal from Mike that he feels threatened by Bob, in which case John's better off going exclusive with Bob.
This is exactly the analogy to keep in mind while choosing between CPA and traditional advertising (or CPM) models. With traditional models, you pay to showcase your offering. The ad-network will get paid by you regardless of whether your ads make you money. Once you've signed up to deliver your ads through them, you have effectively committed your funds. To be fair, it's in the interest of the ad-network to make you money, of course. They know you won't come back if you don't realize revenue. But it's just not a life-or-death issue for them like it is with CPA networks. As an advertiser, I'd be prudent to use the services of a network for which making me money is a "burning" issue, not just a desirable one.
Because the viability of a CPA ad-network is most closely tied to your own success, you should also expect them to invest the most effort in ensuring the best destination web pages for your ads. For this reason, you can also expect the most automation from CPA networks, and you as an advertiser will not have to learn the black art of tweaking any of the gazillion knobs to route your ad correctly. Very often, due to the sheer number of possible ad-destinations, a machine-determined match in real-time is bound to be of better quality than manual specifications. For example, you could have asked for your Canon-A500 ad to appear on pages under photographyblog.com, but it's quite possible that a different website summarizing features of various compact cameras is a much better destination for your ad than a page under photographyblog.com that talks about scenic spots in Nepal.
In any case, regardless of how the match of your ad to a web page is determined, there is bound to be competition for advertising space on that page. In this situation, networks with at least a nominal amount of automatic targeting tend to do the fairest thing. Ads will be algorithmically ranked by relevance to that page and only then will ties be broken by bid price for the contested spot. In some cases, a more sophisticated measure based upon both relevance and bid-price is calculated; but you will be hard-pressed to find a self-respecting automatic-targeting network, whether CPA or CPC (Cost-PerClick), that shows totally irrelvant ads on a page just because the advertiser offered a high bid-price for that spot.
In this scenario, both the CPA and CPC frameworks work in favor of the advertiser because the networks stand to lose by showing an irrelevant ad (No clicks or no actions mean no revenue for the network). No such luck, however, with traditional or CPM advertising because you as an advertiser have to shell out your precious pennies even if your ads for snow boots are shown on Equatorial Guinea tourism pages.
Far from flipping coins to serve its clients, it should now be clear that a CPA network needs to do several things. It needs to collect a vast inventory of destination web-site addresses. It needs to accept and store a large number of ads from advertisers. Every time a web page in its inventory is displayed on a browser, it needs to select the most relevant ads from this bank, resolve ties for spots based primarily on relevance, but also (and only secondarily) on the spot's bid-price, and display the ads. But unlike for CPM and CPC networks it doesn't end here; CPA networks further need to track viewers' actions. If a viewer performs an action that counts, then, and only then, should it bill the advertiser. Clearly, infrastructure to do this is sophisticated, complex and costly. You, as an advertiser, must understand that it is doubly in the interest of the CPA ad-networks to ensure your ads perform the way you want. They want to maximize your revenue because that's the only way they can maximize theirs.
Before wrapping up, let's briefly look at three common myths about CPA.
MYTH 1: In a CPA framework, it is easy to defraud advertisers by generating "fake-actions"
DEBUNKED: Click-fraud is a major problem with CPC advertising and indeed there need be no greater indictment of click-fraud in CPC than a telling statement by Eric Schmidt, the chief of CPC king Google: "...there is a perfect economic solution [to click-fraud] which is to let it happen". There is, in fact, a better solution - CPA. With CPA, advertisers do not pay unless they have a sale, lead, or some other valuable action. Intelligently selected actions are not recorded unless a confirmation page is displayed (at which point your customer has already been "billed"). A fraudster cannot fake such an action without actually being accountable to you.
MYTH 2: In a CPA framework, advertisers don't have any control over where their ads appear.
DEBUNKED: CPA advertisers have at least as much control over where their ads appear as they do with CPC or CPM. In fact, as I described earlier, the CPA network has the most vested interest in ensuring that your ads are well placed. They take your advice on potential ad destinations most seriously because they appreciate that you, as an advertiser, know best about your own domain. Simultaneously, like many CPC networks, they also provide you with the option of leveraging powerful AI algorithms to help you target your ads precisely.
MYTH 3: "The CPA mindset is dangerous. It can kill your business" (quoting Kevin Lee in his ClickZ column)
DEBUNKED: As I have tried to explain, not only is the CPA mindset not dangerous, but it is actually detrimental to your business to not think CPA. To claim that this uniquely customer-empowering natural evolution of advertising can kill your business is regressing back to primitive days. It is like the warning "Sailing past the horizon is dangerous. You could fall off the edge of the earth and kill your passengers."
]]>