• Very interesting
• You need more graphics. I know it's sooo tempting to do much with text, but it's much nicer to look at if you include more graphics. (Update: much better now).
• I don't know how good the math-fu of your audience is, but you try to assume as little as possible. For example when you define Var(X), show some graphs from curves with high and low variance. Much easier to imagine for a non-math-geek
• Since you declared your talk to be a tutorial, I'd suggest picking one example, and apply the various techniques against that same example. Give the people something familiar to cling to if they feel lost
• 3 hours is much. Very much. Perhaps too much. My usual lectures (as a student) are 1.5h, and I'm nearly alway glad when they're over. So whatever you do, don't talk for three hours. Do something else in the mean time (show some graphs and shut up for a few minutes, or whatever; I don't really have a good idea what to do, just what not to do)

Graphics is something I've heard consistently, and I'm going to improve that.

My audience have self-selected themselves to go to a 3 hour tutorial about how to use statistics to optimize their website. I'm going to assume that they are at least comfortable with math. That said, I've pushed most of the heavy math off to the last third of the talk, and I've made it clear that they don't have to understand that material to run successful A/B tests. So I'm hoping that my audience will divide into three groups there. People for whom it is review and for which rushing is fine. People who want to learn who can go back and read it again. People who realize they can skip that and therefore don't mind being lost.

In the talk I'll be reinforcing that by making it clear that this is a deeper look and is not strictly necessary.

The schedule is by O'Reilly. There is a half-hour break in the middle. I don't have resources to do any real exercises. Yes, I'm painfully aware that this will be hard on the audience. It is unfortunately worse on me. :-(

In the 'Show Me' slide there's a superfluous 'I'

. Starting with the slide 'Programming API' it was a bit confusing. Because at first I thought version meant version of the program code. Also a lot of "unspecific" sentences like 'do the appropriate thing' 'This does what it says'. Probably that works better when a presenter fills in the blanks.

Why does get_or_create_test_version need a person_id to create a test version?

"It finds the current version". In what way is a version current? Shouldn't there be 2 or more current versions?

"If none is found it chooses a random version... and saves it". Why save that random version. Shouldn't it be already stored in some version pool (otherwise it couldn't be selected. Or does your library *generate* random versions?

My main gripe here seems to be that the programming API is thrown in too early, I would have liked data definitions before API definitions. Then show the API and give an example how it is used to generate a test with two versions and display the test for a specific user. But maybe this is just a sign of my ignorance of the subject of your presentation or a sing of my brain not working

Got to slide 40 where I needed to think a while before I understood what 'changing the mix they get' means. '... while the test is running'. Probably not a problem at the presentation since you will be filling in the blanks

Got no further because I need to work for my day job some more, but hope this helps a little

You're right that I will be filling in a lot of those blanks.

To answer your question, we are assigning each person a test version, and then tracking how they do. This is the API you call to find out what version of the page this user gets. If the user has never been assigned a test version, then they will be assigned one. The calling code does not care whether the version was newly created. But you definitely need the person in there because the test assignment is tied to the person.

About the order I gave the explanation in, I've tried explaining it both ways and had people draw blanks. Conversely I've had people find it too obvious to talk about either way.

Still I'll go through and review that section with an eye to making it clearer.

Ah, much clearer now. My problem here was that you didn't say that you were only showing the API for a central but incomplete part of the A/B-test framework, one could say the controller interface of the MVC model. The management of tests and versions, their mapping from name to webpage, the logging (for example log_test_activity()), and all the statistical analysis is (in a strict view) part of the API too.

Just mentioning in that slide that "that is the function that selects which version of a test to show a person on the website" would provide context and make this slide so much easier to understand. That sentence may be obvious to a smal l or large part of your audience but would have helped the others to keep up (at least it would have helped me, 100% of this test group ;-). It is also not the whole truth about the routine, but its possible use in the statistics phase to map users to versions can be explained later

"don't have time to do real exercises": Do you mean test presentations or Moritz suggestion to provide examples? If the latter, you don't need *real* exercises/examples. A hypothetical example can be trivial and senseless, it is a way to ground the theory in the practical.

Look at the slide "Behind the scenes". Apart from the first sentence "This does what it says" (which has absolutely no information value at all) this slide could have been replaced with an example where you provide the same information in the context of your example.

Let's say you have (on a previous slide) given the example of a english to suaheli translation page and you want to test it with versions with green and blue background color.

----------------------------------
Peters first visit
------------------
Peter visits a page -> web server calls get_or_create_test_version(<pa
+ulsid>,'EtoS',['green','blue']);<p>

New User? [small picture of file cabinet with a arrow pointing to a '?
+'] YES.
[small picture of a dice or coin toss] -> random choice of 'green' or 
+'blue'. Let's say 'green'. ->
[picture of string 'peter has green' with arrow to file cabinet ]. 

$version='green'

[ small picture of a green cube which looks like a webpage ]

----------------------------
Peters next visits
----------------
Peter visits a page --> web server calls get_or_create_test_version(<p
+aulsid>,'EtoS',['green','blue']);

New User? [picture of file cabinet with an arrow to string "peter has 
+green"] NO.

$version = 'green'

[ small picture of a green cube which looks like a webpage ]
------------------------
[download]

Yes, this is trivial (and could be improved a lot). But this gives the same information as the replaced slide and because of the pictures and the concreteness is faster and easier to understand and remember.

I don't think that a picture on every slide is beneficial. This will distract many with questions like 'What has that picture to do with the message of this slide?', especially with long presentations. I would put pictures into introducory, title/heading/opening/closing slides (slides 0 and 47 for example) and easy/low complexity slides (slide 4). On all other slides there should be a picture only if it either:
1) contributes to the information on the page (slide 5 is a positive example), or
2) loosens up the presentation by showing a cartoon, a famous saying or a funny picture that has some relevance to this slide. This cartoon or picture must be the first thing on the slide (because it is eye-catching, everyone will read or look at it first) and you have to give the listeners time to view and possibly laugh about this picture.

Disclaimer ;-): All of this IMHO. And apart from the above I like your presentation and find it interesting, but you wanted to know about the bad not the good parts.

Hello,

Thanks for sharing! Very interesting.

Havent finished yet, but:

Really non issues but would help with readability. Again, thanks. Great content.

trwww

Thank you for the feedback, I've updated my slides and rewrote that one to be clearer.

The big change is that it now has lots and lots of pictures. :-)

Now there are too many pictures. :-) Just kidding.

---

I'm not really a human, but I play one on earth CandyGram for Mongo

As for the talk itself, that is kind of a daze for me. I pretty much was going on fumes. And I know I was going fairly quickly because what took me 3.5 hours to do on my own finished in 2.5 hours with questions. It is hard to judge what an audience is thinking. If anyone is wondering how it went, you'd be better off at this point asking vroom. He was present and experienced it from a point of considerably more sleep than I did. :-)