Tales of the Sausage Factory

*Packingham* And The Public Forum Doctrine Part I: Implications For Copyright.

On Monday June 19, the Supreme Court issued two significant First Amendment decisions. Most of the press attention went to Matal v. Tam aka “The Slants'” case. But the far more significant case for my little neck of the woods was Packingham v. North CarolinaBecause Packingham focused on criminal law, and did not have anything to do with the Washington Redskins keeping or changing their name, it garnered relatively little attention. But Packingham has much more importance for the future of the First Amendment online by recognizing the primary First Amendment right of subscribers to access broadband platforms and content. Indeed, Justice Kennedy’s paean to the Internet as the modern public square echoes themes from the more “Madisonian” view of the First Amendment expounded by scholars such as Cass Sunstein (and prompted alarm from Alito, Roberts and Thomas in concurrence).

 

This has significant implications for all the crap the Digital Millenium Copyright Act (DMCA) has done to make it easy to kick users offline (and the whole future of “graduated response”/”3 strikes”) and the existing and fairly abusive notice and takedown regime (and efforts to extend it further). It may also have significant implications for the First Amendment argument over broadband, net neutrality and the future of regulation of online platforms such as Facebook, but I will save that for Part II.

 

I unpack all this below . . .

Read More »

Posted in Censorship Public and Private, Fighting the IP Mafia, Tales of the Sausage Factory | Tagged , , , , , , | 1 Comment

Tales of the Sausage Factory

NCTA Agrees Title II Virtuous Cycle Totally Working; Or, Pai’s Economics v. the Actual Real World.

Last week, NCTA, the trade association for the industry formerly known as cable, posted this amazing graph and blog post showing that the “virtuous cycle” the FCC predicted would happen when it adopted the open Internet rules (aka Net Neutrality) back in December 2010. Indeed, as the NCTA graph (based on the latest Akamai State of the Internet Report) shows, the average speed of broadband connections has not only continued to rise since the FCC first adopted net neutrality rules in 2010, but the rate of increase has accelerated since the FCC adopted the Title II Reclassification Order in February 2015. Finally, as NCTA also points out, in the approximately 10 years since the FCC first began to enforce net neutrality through the “Internet Policy Statement” and the Comcast/Bittorrent Complaint, the cost of moving bits from their source to your home has dropped 90% on a per bit basis. (Whether we are actually still paying too much because of our lack of competition in the broadband market is something of a different question.)

 

Perhaps unsurprisingly, this matches the findings from Free Press’ Dr. Erik Turner in this massive and meticulously documented report, “Broadband Investments And Where To Find Them.” But it’s still nice to see NCTA confirm it. One of the advantages of having blogged on net neutrality for 10 years is I can point to things like this 2006 blog post and say: “Hey, I totally predicted that. Glad to see things working as I predicted they would.” This contrasts with the net neutrality haters, who as far back as 2006 that predicted that preventing ISPs from discriminating and prioritizing traffic would result on average broadband quality getting consistently worse a bandwidth kept treating the Internet “like a truck you can just load things on” instead “of a series of tubes.”

 

 

So why did the self-appointed experts get it so wrong? And why do they still fixate on criteria like “ISP CAPEX” that neither Congress nor anyone outside the economics world cares about (and which a reviewing court utterly will not give a crap about) if better faster broadband is getting deployed as we all predicted and Congress directed?

 

 

The answer boils down to the old cliche: “Among economists, the real world is often a special case.” So while all of us out here in the real world focus on things like “hey, is broadband actually getting deployed, and is it getting better and faster and stuff so we can do all the things that make better faster broadband so critical in everyone’s lives these days,” economists poo-poo such concerns as being part of an “economics free zone.” Questioning this navel gazing in Econ Cloud Cuckoo Land will evoke sneers about how silly you must be for not understanding why the actual real world is irrelevant to the purity and wonderfulness of “real” economics. For some odd reason, a lot of folks eat this superior attitude up with a spoon and fail to ask the follow up question like “you know you didn’t actually address the substance of the argument, right?”

 

Anyway, I will below unpack all of this by: (a) reviewing what we actually predicted about the virtuous cycle; (b) reminding folks about the predictions of doom and gloom from the haters in Econ Cloud Cuckoo Lad (that’s a literary reference, btw, for when the usual suspects want to get all fake outragey to avoid dealing with substance); (c) reviewing why the evidence is consistent with the pro-Net Neutrality prediction and falsifies the anti-Net Neutrality prediction; and (d) why this means that if Pai tries to base his roll back of Title II/net neutrality by embracing the Singer/USTA CAPEX argument and ignoring all the other evidence, he is going down in flames in the D.C. Circuit.

 

(I would love include a section on what ISP CAPEX actually should look like, which casts further doubt on the question of the relevancy of any modest drop in ISP CAPEX over time as a useful measure, but I’m gonna have to save that for a later follow up.)

Read More »

Posted in Series of Tubes, Tales of the Sausage Factory | Tagged , , , , | 4 Comments

Tales of the Sausage Factory

Will Pai “Pull A Putin” And Hack the FCC Process? Or Will He Get Over Himself and Start Acting Like The Chairman?

In my 20+ years of doing telecom policy, I have never seen a Chairman so badly botch a proceeding as Chairman Ajit Pai has managed to do with his efforts to repeal Net Neutrality. For all the fun that I am sure Pai is having (and believe me, I understand the fun of getting all snarky on policy), Pai’s failure to protect the integrity of the process runs the serious risk of undermining public confidence in the Federal Communications Commission’s basic processes, and by extension contributing to the general “hacking of our democracy” by undermining faith in our most basic institutions of self-governance.

 

Yeah, I know, that sounds over the top. I wish I didn’t have to write that. I also wish we didn’t have a President who calls press critical of him “the enemy of the American people,” triggering massive harassment of reporters by his followers. What both Trump and Pai seem to fail to understand is that when you are in charge, what you say and do matters much more than what you said and did before you were in charge. You either grow up and step into the challenge or you end up doing serious harm not only to your own agenda, but to the institution as a whole. Worse, in a time when the President and his team actually welcomed Russia’s “hacking” of our election, and remain under suspicion for coordinating with Russia for support, Pai’s conduct creates concern and distrust that he will also “pull a Putin” by welcoming (or worse, collaborating with) efforts to de-legitimize the FCC’s public comment system and hack the public debate around net neutrality generally.

 

Fortunately, as I told former Democratic FCC Commissioner Julius Genachowski when he was in danger of making the FCC’s process a laughingstock in the public eye, Pai can still recover and rescue himself and the FCC from his self-destructive conduct. Instead of calling his critics enemies of capitalism and free speech, instead of obsessing about his own hurt feelings while displaying a troubling indifference to identity stealing bots filing comments that support his own proposal and failing to follow up on his own claims that the FCC comment system suffered a critical cyber-attack – Pai needs to follow in the footsteps of Michael Powell, Kevin Martin and Tom Wheeler when they faced similar insults (and in Powell’s case, racial slurs). Welcome robust public debate and criticism, condemn the actually illegal hacking used by his supporters, and stop whining about his own hurt feelings. Michael Powell managed to take being called a War Criminal and son of a war criminal for supposedly allowing the press to sell us on the Iraq War, as well as the same kind of racist bullshit that Pai or any other prominent person of color sadly has to endure in an America where racists feel increasingly emboldened. Pai can chose to step up in the same way his Republican and Democratic predecessors did, or continue to contribute to the overall erosion of trust in our institutions of self-governance generally and his handling of the FCC specifically.

 

I unpack all this below . . .

Read More »

Posted in "A Republic, if you can keep it", How Democracy Works, Or Doesn't, Life In The Sausage Factory, Series of Tubes, Tales of the Sausage Factory | Tagged , , , , , , | 1 Comment (Comments closed)

Tales of the Sausage Factory

Welcome Back to the Net Neutrality Fight Summer Blockbuster Reboot!

Hi everyone! Back from a 3 month sabbatical and my Mom’s heart surgery, and just in time for the nth+1 replay round on Network Neutrality. As with so many things, I can’t believe we are going to reboot this franchise once again and run through pretty much the same arguments. But as with repeal of Obamacare, Republicans would rather focus themselves on undoing Obama’s legacy rather than moving on and getting stuff done. Since they run the show, we play this game again.

 

Regular followers of this blog will know I have been fighting the net neutrality fights since they began back in 1998 (when it was the “open access” fight and the telcos were on our side). I have seen a steady stream of victories and defeats. Time and again, we have found ourselves backed into a corner and had to rally when everything seemed hopeless. However, as I explained back in 2010, there are reasons why network neutrality refuses to die, but that doesn’t mean we’ll win (this round) either. So, in the spirit of movie reboots and sequels, I will quote Captain Kirk to Captain Picard: “I take it the odds are against us and the situation is grim . . . Sounds like fun!”

 

While I had certainly hoped the Republicans would see reason — Pai has made it clear that he is as obsessed with exterminating net neutrality and every other pro-competitive and pro-consumer policy at the FCC. Pai is obsessed with demolishing every single accomplishment of Wheeler’s as Kahn was to have his revenge on Kirk (which did not, in fact, work out very well for Kahn).  But Pai, and Blackburn and Senator Lee go beyond the usual Obama/Wheeler derangement syndrome (“Wheeler, hates it precious!“) This is full on Davros and the Daleks utter willingness to destroy reality.

 

Now I’ve heard people ask: “But the Republicans control the FCC. They control both houses of Congress. They are determined to ignore the millions of people who have already made their opposition plain, and ignore all the mountains of evidence that sits before them. What can we possibly do?”

 

Well, I have a message for Chairman Davros and his army of industry Daleks.

 

Stay tuned . . . .

Posted in "A Republic, if you can keep it", Cable, Series of Tubes, Tales of the Sausage Factory | Comments closed

Tales of the Sausage Factory

My Letter To Trump On Why I Need to Hear Him Say: “Anti-Semitism Is Bad.”

Dear Mr. President.

 

Why is it so hard for Donald Trump to say “Anti-Semitism is bad, and the US government will protect all people from hate crimes no matter what their race or religion”? This is really getting deeply troubling.

 

Yes, I get it. Jared, Ivanka, the grandkids. You love Israel. You get on great with Bibi. You have lots of Jewish friends. I’m sure Trump Tower makes the best falafel and humous on Israel Independence Day, and the best chopped liver on Rosh Hashanah.

 

But for some reason, in several consecutive press conferences, the rather simple and straightforward statement that “Anti-Semitism is bad. The government of the United States will not tolerate threatening Jews with violence, vandalizing synagogues or Jewish institutions, or otherwise treating Jews differently than anyone else,” or words to that effect, have not come out of your mouth. And that is a real problem for me.

 

I’m an Orthodox Jew. I’m generally supportive of the State of Israel. And, if Trump Tower had a hechsher, I’m sure I’d love your felafel or chopped liver. I’m also an American, and very proud of that. I have always been proud of being an American citizen. I have thrilled with pride when I testify before Congress on super boring telecommunication policy that here I am, wearing my kippah, being all open Jew person, and not here just to testify on Israel of some other Jewish topic. I walk through the “Halls of Power” not as a supplicant petitioning for favors — as my ancestors in Europe and the Middle East were forced to do — but as a proud citizen exercising my First Amendment right to “petition the government for redress of grievances.”

 

I have spent the bulk of my professional life in public policy, because I passionately believe in the promise and ideals of the United States of America.

 

And yes, you are my President. True, I voted against you. I oppose just about every policy decision you have made so far. But you are still the man who was elected President of the United States under the rules of the Constitution. That makes Donald Trump the President of the United States, and therefore my President.

 

So please understand. I really, really need to hear my President say: “The President of the United States denounces anti-Semitism. You, Harold Feld, have the same rights as every other American.” Not “hey, I’ve got Jewish grandkids” or “I’m the least Anti-Semitic person ever.”

 

I know I’m not the only one who probably needs to hear that explicitly. I know in these times that other people are under attack for their religion, for their race, for their gender or sexual orientation. I’m pretty sure they want to hear it explicitly from their President (whether they like him or not, whether they believe him or not). But I can only speak personally for me. I can tell you, as an American and Orthodox Jew, that I need to hear from my President that I am still an American who just happens to be Jewish — not a Jew who happens to live in America.

 

If you aren’t sure exactly what to say, here are the words that our first President, George Washington, used to reassure the Jews of Newport Rhode Island. At the time, there was not a single country in the world where Jews enjoyed equal rights as citizens. The best Jews could hope for was “toleration,” which could be withdrawn at any time. President Washington therefore reassured the Jews of America:

 

“It is now no more that toleration is spoken of, as if it was by the indulgence of one class of people, that another enjoyed the exercise of their inherent natural rights. For happily the Government of the United States, which gives to bigotry no sanction, to persecution no assistance, requires only that they who live under its protection should demean themselves as good citizens, in giving it on all occasions their effectual support.”

 

It would mean an awful lot to hear you quote those words, or say something similar.

 

Sincerely,
An American Citizen who happens to also be an Orthodox Jew

Posted in "A Republic, if you can keep it", How Democracy Works, Or Doesn't, I Fear These Things, Tales of the Sausage Factory | Tagged , , | Comments closed

Tales of the Sausage Factory

Trump Keeps Us All Guessing On Telecom.

Usually in January, especially with a new Congress of new term, I like to try to do a “this year in telecom” preview. Hell, who doesn’t? (I mean, who in Telecom Policyland doesn’t. The answer for normal people is: “no one.”) But this year I can’t.

 

Oh, I can list all the issues we’ve been arguing over the last few years and guarantee we’re going to re-litigate them. We’ve already seen most of the ISP industry (joined by the Ad industry) push back on the privacy rules adopted last October.  We’ve seen a bunch of the industry submit their wish list for deregulation as part of the bienniel telecom regulatory review. And with Rep. Marsha Blackburn (R-TN) now Chair of the Telecom Subcommittee, we can expect lots of action on the Hill side on everything from FCC process reform to Telecom Act re-write. But the Trump Administration itself — its priorities, its possible pick for FCC Chair, and its general direction on telecom policy — remain as much a mystery as when I wrote about it last month.

 

Read More »

Posted in Life In The Sausage Factory, Tales of the Sausage Factory | Tagged , , , , | Comments closed

Tales of the Sausage Factory

Is Net Neutrality (And Everything Else) Not Dead Yet or Pining For the Fjords? Contemplating Trump’s Telecom Policy.

The election of Donald Trump has prompted great speculation over the direction of telecom policy in the near future. Not surprisingly, everyone assumes that the primary Republican goal will be to completely roll back net neutrality and just about every other rule or policy adopted by the Wheeler FCC — perhaps even eliminating the FCC altogether or scaling back it’s authority to virtual non-existence. Why not? In addition to controlling the White House, Republicans have majorities in the Senate and the House.  Jeff Eisenach, the head of Trump’s FCC transition team (now called “Landing Teams”), has been one of the harshest critics of the FCC under both Wheeler and Genachowski. So it is unsurprising to see a spate of articles and blog posts on the upcoming death of net neutrality, broadband privacy, and unlicensed spectrum.

 

As it happens, I have now been through two transitions where the party with the White House has controlled Congress. In neither case have things worked out as expected. Oh, I’m not going to pretend that everything will be hunky-dory in the land of telecom (at least not from my perspective). But having won things during the Bush years (expanding unlicensed spectrum, for example), and lost things in the Obama years (net neutrality 2010), I am not prepared to lay down and die, either.

 

Telecom policy — and particularly net neutrality, Title II and privacy — now exists in an unusual, quantum state that can best be defined with reference to Monty Python. On the one hand, I will assert that net neutrality is not dead yet. On the other hand, it may be that I am simply fooling myself that net neutrality is simply pining for the fjords when, in fact, it is deceased, passed on, has run up the curtain and joined the choir invisible.

 

I give my reasons for coming down on the “not dead yet” side — although we will need to work our butts off to keep from getting clopped on the head and thrown into the dead cart. I expect the usual folks will call me delusional. However, as I have said a great deal over the years: “If I am delusional, I find it a very functional delusion.”

 

More below . . . .

Read More »

Posted in Life In The Sausage Factory, Tales of the Sausage Factory | Tagged , , , , , , , , , , | Comments closed

Tales of the Sausage Factory

The George Washington Pledge: “To Bigotry No Sanction, To Persecution No Assistance.”

I’m starting what I call the George Washington Pledge.

 

THE GEORGE WASHINGTON PLEDGE

“I pledge to give to bigotry no sanction, to persecution no assistance. I pledge to work toward a world where everyone may sit under their own vine and fig tree, and there shall be none to make them afraid. A world that scatters light and not darkness in our paths, and makes us all in our several vocations useful here, and in due time and way everlastingly happy.”

 

Where did that come from, what does it have to do with George Washington and don’t I know that George Washington was a bigot who kept slaves? To answer the second question first, yes. I know that it is one of the great and cruel tragedies of history that George Washington himself, while expressing these concepts, was committing the ultimate bigotry and persecution by holding slaves and asserting that those of African descent were not fully human. Nevertheless, while this pledge made by the First President of the United States has never been fulfilled, it time we committed to making it true.

 

We live now in a time when it is the duty of those of us committed to the success of the American Experiment in self-rule to remember the promises and values which the founders of our country made the foundation of governance. Whatever their past success, whatever the sincerity of those who wrote the words, it falls on us to do our part to make these foundational values real. To quote the words of our first President: “If we have wisdom to make the best use of the advantages with which we are now favored, we cannot fail, under the just administration of a good Government, to become a great and a happy people.”

 

So where do the words of the George Washington Pledge come from? And what do I mean when I commit myself to it? See below . . . Read More »

Posted in "A Republic, if you can keep it", How Democracy Works, Or Doesn't, I Fear These Things, Tales of the Sausage Factory | Tagged | 2 Comments (Comments closed)

Tales of the Sausage Factory

Are Police Jamming Cell Phones At Standing Rock Protest? The FCC Should Investigate.

Given the lack of coverage in mainstream media, you might not have heard about the ongoing protest against the construction of the Dakota Access Pipeline immediately upstream from the Standing Rock Sioux reservation aka #NoDAPL. You can find some good statistics on the pipeline and number of arrests associated with the protest here. Setting aside my personal feelings about democracy, freedom to peacefully protest, and how the Sioux concerns seem rather justified in light of the Alabama pipeline explosion, this has now raised an interesting communications issue that only an FCC investigation can solve. Are police jamming, or illegally spying, on communications at the protest and associated Sacred Stone Camp?

 

Over the last week, I have seen a number of communications from the protest about jamming, particularly in the period immediately before and during the Thursday effort by police to force protesters off the land owned by Dakota Access Pipeline. In addition, this article in Wired documents why tribal leaders connected with the tribal telecom provider, Standing Rock Telecom, think they are being jammed. I’ve had folks ask to speak to me using encrypted channels for fear that law enforcement will use illegal monitoring of wireless communications. As this article notes, there are a number of telltale signs that law enforcement in the area have deployed IMSI catchers, aka Stingrays, to monitor communications by protesters. However, as I explain below, proving such allegations — particularly about jamming — is extremely difficult to do unless you are the FCC.

 

Which is why the FCC needs to send an enforcement team to Standing Rock to check things out. Given the enormous public interest at stake in protecting the free flow of communications from peaceful protests, and the enormous public interest in continuing live coverage of the protests, the FCC should move quickly to resolve these concerns. If law enforcement in the area are illegally jamming communications, or illegally intercepting and tracking cell phone use, the FCC needs to expose this quickly and stop it. If law enforcement are innocent of such conduct, only an FCC investigation on the scene can effectively clear them. In either case, the public deserves to know — and to have confidence in the Rule of Law with regard to electronic communications.

 

More below . . . .

Read More »

Posted in "A Republic, if you can keep it", Censorship Public and Private, How Democracy Works, Or Doesn't, Spectrum, Tales of the Sausage Factory | Tagged , , , , | Comments closed

Inventing the Future

Discovery Part 2: How

 “The Universe is made of stories, not of atoms.” -Muriel Rukeyser

Last time, I discussed why we offer suggested locations to teleport to, and the 5 W’s of the user interaction for suggestions. This time I’ll discuss how we do that, with some level of technical specificity.

Nouns and Stories

Each suggestion is a compact “story” of the form: User => Action => Object. Facebook calls these “User Stories” (perhaps after product management language), and linguists refer to “SVO” sentences. For example, Howard => snapshot => Playa, which we display as “Howard took a snapshot in Playa”. In this case Playa is a Place in-world. The Story also records the specific position/orientation where the picture was taken, and the picture itself. Each story has a unique page generated from this information, giving the picture, description, a link to the location in world, and the usual buttons to share it on external social media. Because of the metadata on the page, the story will be displayed in external media with the picture and text, and clicking on the story within an external feed will bring them to this page.

snapshot-howard-playa

The User is, of course, the creator of the story. The user has a page, too, which shows some of their user stories, and any picture and description they have chosen to share publicly. If allowed, there’s a link to go directly to that user, wherever they are.

user-card-howard

For our current snapshot and concurrency user stories, the Object is the public Place by which the user entered. More generally, it could be any User, Place, or (e.g., marketplace) Thing. These also get their own pages.

place-card-playa

The “feed” is then simply an in-world list of such Stories.

Control

Analogously to any computer on a network and registering with ICANN, a High Fidelity user may create places at an IP address or even a free temporary place name, or they can register a non-temporary name. Places are shown as suggestions to a user only when they are explicitly named places, with no entry restrictions, matching the user’s protocol version. (When we eventually have individual user feeds, we could consider a place to be shareable to a particular user if that logged in user can enter, rather than only those with no restrictions.)

Snapshots are shown only when explicitly shared by a logged-in user, in a shareable place.

snapshot-review

Scale

At metaverse scale, there could be trillions of people, places, things, and stories about them. That’s tough to implement, and tough for users to make use of the firehose of info. But now there isn’t that many, and we don’t want to fracture our initial pioneer community into isolated feeds-of-one. So we are designing for scale, but building iteratively, initially using our existing database services and infrastructure. Let’s look first at the design for scale:

First, all the atoms of this universe – people, places, things, and stories – are each small bags of properties that are always looked up by a unique internal identifier. (The system needs to know that identifier, but users don’t.) We will be able to store them as “JSON documents” in a “big file system” or Distributed Hash Table. This means they can be individually read or written quickly and without “locking” other documents, even when there are trillions of such small “documents” spread over many machines (in a data center or even distributed on participating user machines). We don’t traverse or search through these documents. Instead, every reason we have for looking one up is covered by something else that directly has that document’s identifier.

(There are a few small exceptions to the idea that we don’t have to lock any document other than the one being looked up. For example, if we want to record that one user is “following” another, that has to be done quite carefully to ensure that a chain of people can all decide to follow the next at the same time.)

There are also lists of document identifiers that can be very long.  For example, a global feed of all Stories would have to find each Story one or more at a time, in some order. (Think of an “infinitely scrollable” list of Stories.) One efficient way to do that is to have the requesting client grab a more manageable “page” of perhaps 100 identifiers, and then look up the document on however many of those fit on the current display. As the user scrolls, more are looked up. When the user exhausts that set of identifiers, the next set is fetched. Thus such “long paged lists” can be implemented as a JSON document that contains an ordered array of a number of other document identifiers, plus the identifier for the next “page”. Again, each fetch just requires one more document retrieval, looked up directly by identifier. The global feed object is just a document that points to the identifier of the “page” that is currently first.  Individual feeds, pre-filtered interest lists, and other features can be implemented as similar long paged lists.

However, at current scale, we don’t need any of that yet. For the support of other aspects of High Fidelity, we currently have a conventional single-machine Rails Web server, connected to a conventional Postgres relational database. The Users, Places, and Stories are each represented as a data table, indexed by identifier.  The feed is a sorted query of Stories.

We expect to be able to go for quite some time with this setup, using conventional scaling techniques of bigger machines, distributed databases, and so forth.  For example, we could go to individual feeds as soon as there are enough users for a global feed to be overwhelming, and enough of your online friends to have High Fidelity such that a personal feed is interesting, This can be done within the current architecture, and would allow a larger volume of Stories to be simultaneous added, retrieved, scored, and sorted quickly.  Note, though, that we would really like all users to be offered suggestions — even when they choose to remain anonymous by not logging in, or don’t yet have enough experience to choose who or what to follow. Thus a global feed will still have to work.

Scoring

We don’t simply list each Story with the most recent ones first. If there’s a lot of activity someplace, we want to clearly show that up front without a lot of scrolling, or a lot of reading of place or user names. For example, a cluster of snapshots in the feed can often make it quite clear what kind of activity is happening, but we want the ordering mechanism to work across mixes of Stories that haven’t even been conceived of yet.

Our ordering doesn’t have to be perfect – there is no “Right Answer”. Our only duty here is to be interesting. We keep the list current by giving each Story a score, which decays over time. The feed shows Stories with the highest scores first. Because the scores decay over time, the feed will generally have newer items first, unless the story score started off quite high, or something bumped the score higher since creation. For example, if someone shares a Story in Facebook, we could bump up the score of the Story — although we don’t do that yet.

Although we don’t display ordered lists of Users or Places, we do keep scores for them. These scores are used in computing the scores of Stories.  For example, a snapshot has a higher initial score if it is taken in a high scoring Place, or by a high scoring User. This gives stories an effect like Google’s page ranking, in which pages with lots of links to them are listed before more obscure pages.

To keep it simple, each item only gets one score. While you and I might eventually have distinct feeds that list different sets of items, an item that appears in your list and my list still just has one score rather than a score-for-you and different score-for-me. (Again, we want this to work for billions of users on trillions of stories.)

To compute a time-decayed score, we store a score number and the timestamp at which it was last updated.  When we read an individual score (e.g., from a Place or User in order to determine the initial score of a snapshot taken in that Place by that User), we update the score and timestamp.  This fits our scaling goals because only a small finite number of scores are updated at a time. For example, when the score of a Place changes, we do not go back and update the scores of all the thousands or millions of Stories associated with that Place. The tricky part is in sorting the Stories by score, because sorting is very expensive on big sets of items. Eventually, when we maintain our “long paged lists” as described above, we will re-sort only the top few pages when a new Story is created. (It doesn’t really matter if a Story appears on several pages, and we can have the client filter out the small numbers of duplicates as a user scrolls to new pages of stories.) For now, though, in our Rails implementation, a new snapshot causes us to update the time-decayed score for each snapshot in order, starting from what was highest scoring. Once a story’s score falls below a certain threshold, we stop updating.  Therefore, we’re only ever updating the scores of a few days worth of activity.

Here are our actual scoring rules at the time I write this. There’s every chance that the rules will be different by the time you read this, and like most crowd-curation sites on the Web, we don’t particularly plan to update the details publicly. But I do want to present this as a specific example of the kinds of things that affect the ordering.

  • We only show Stories in a score-ordered list. (The Feed.) However, we do score Users and Places, because their scores are used for snapshots. We do this based on the opt-outable “activity” reporting:
    • Moving in the last 10 seconds bumps the User’s score by 0.02.
    • Entering a place bumps the Place’s score by 0.2.
  • Snapshot Stories have an initial score that is the decayed average of the User and Place – but a minimum of 1.
  • Concurrency Stories get reset whenever anyone enters or leaves, to a value of nUsersRemaining/2 + 1.
  • All scores have a half-life of 3 days on the part of the score up to 2, and 2 hours for the portion over 2. Thus a flurry of activity might spike a user or place score for a few hours, and then settle into the “normal high” of 2.  This “anti-windup” behavior allows things to settle into normal pretty quickly, while still recognizing flash mob activity.

 

For example, under these rules, one needs to move for about 3:20 minutes / day to keep your score nominally high (2.0).  More activity will help the snapshots you create during the activity, but only for a while, and snapshots the next day will only have an nominally high effect.

As another example of current rules, an event with 25 people will bump a place score by 5:

  • If it started at 2, it will back down to 4.5 in two hours, 2.5 in six hours, and back to 2 in 10 hours.
  • If it started at 0, it we be at 3.5 in two hours, and then roughly as above.

Search

We currently search the filter on the client, filtering from the 100 highest scoring results that we receive from the server. Each typed word appears exactly (except for case) within a word of the description or other metadata (such as the word ‘concurrency’ for a concurrency story). There is no autocorrect nor autocomplete, nor pluralization nor stemming. So, typing “stacks concurrency” will show only the concurrency story for the place named stacks. “howard.stearns snapshot” will show only snapshots taken by me.

When the volume of data gets large enough, I expect we’ll add server-side searching, with tags.

Conclusion

We feel that by using the “wisdom of crowds” to score and order suggestions of what to do, we can:

  • Make it easy to find things you are interested in
  • Make it easy to share things you like
  • Allow you to affirm others’ activities and have yours affirmed
  • Connect people quickly
  • Create atomic assets that can be shared on various mediums

In doing so, we hope to create a better experience by bringing users to the great people and content that are already there, and encourage more great content development.

Posted in Inventing the Future, Software | Comments closed
  • John Sundman’s Books

    Acts of the Apostles

    Acts of the Apostles

    A thriller about nanomachines, neurobiology, Gulf War Syndrome, and a Silcon Valley messiah.

    "...a book infused with a sensibility that you don't normally expect a 'hard science fiction' novel to have: real emotions, real heartbreak and a real sense of the craziness at the core of the human condition."

    —Andrew Leonard, Salon.com

    Buy or Download

    Cheap Complex Devices

    Cheap Complex Devices

    An anthology of the winners of the inaugural Hofstadter Prize for Machine-Written Narrative, with a preface by the editor and an introduction by the Hofstadter Prize Committee

    "Cheap Complex Devices is astonishing, on just about every level a book can be astonishing."

    —Rusty Foster, Kuro5hin.org

    Buy or Download

    The Pains

    The Pains

    In Freemerica, where Orwell's 1984 is fused with Ronald Reagan's 1984, a young monk tries to save the world from disintegration.

    "All three of Sundman's books are somewhere between excellent and brilliant. ... The Pains touches upon the key issues of our time: it is a book which is philosophical to the point of being mystical."

    —Michael Allen, Grumpy Old Bookman

    Buy or Read Online

    Creation Science

    Creation Science

    conspiracy, duplicity, double-crosses, dispensational Christian fascism, misunderstandings, confusions, car crashes, megalomaniacal villains (in and out of government), explosions, gunplay, Russian Mafias, neuroscience, coincidence, mysterious islands not far from Cape Cod, information theory, love, regret, remorse, nostalgia and sex.

    Published by late spring 2010, if not sooner. Pre-orders much appreciated.

    Pre-order Now

  • Connect With Us

    Follow Wetmachine on Twitter!

Username
Password

If you do not have an account: Register