Projects

Cancer clinical trials and the problem of low patient accrual

Inspired by this contest to come up with ideas to increase the low amount of patient accrual for cancer clinical trials, I decided to look more into the data. Bold, by the way, is one of my all time favorite books, and was co-authored by the creator of the herox.com website, the xprize Foundation, and co-founder of Planetary Resources: Peter Diamandis. Truly someone to look up to.

Anyways, the premise of the contest is that over 20% of cancer clinical trials don’t complete, so the time and effort spent is wasted. The most common reason for this termination is the clinical trial not being able to recruit enough patients. Just how common is the low accrual reason though? And are there obvious characteristics of clinical trials that can help us better predict which ones will complete successfully, and what does that suggest about building better clinical trial protocols? I saw this as an opportunity to explore an interesting topic, while playing around with the trove of data at clinicaltrials.gov and various data analysis python libraries: seaborn for graphing, scikit-learn for machine learning, and the trusty pandas for data wrangling.

Basic data characteristics

I pulled the trials for a handful of the cancers with the most clinical trials (completed, terminated, and in progress), got around 27,000 trials, and observed the following:

  • close to 60% of the studies are based in the US*
location_distribution

*where a clinical trial is “based” can mean where the principal investigator (the researcher who’s running the clinical trial) is based. clinicaltrials.gov doesn’t give the country in which the principal investigator’s institution is in, so as a proxy, I used the country which had the largest number of hospitals the study could recruit patients at.

  • almost 25% of all US based trials ever (finished and in progress) are still recruiting patients

overall_status_distribution

  • of those trials that are finished and have results, close to 20% terminated early, and 80% completed successfully (which matches the numbers the contest cited)

finished_status_distribution

  • almost 50% of all US based trials are in Phase II, almost 25% are in Phase I

phase_distribution

  • and interestingly, the termination rate does not differ very significantly across studies in different phases

status_by_phase

Termination reasons

Next, I was interested in finding out just how common insufficient patient accrual was as a trial termination reason vs. others reasons. This was a little tricky, as clinicaltrials.gov gives principal investigators a free-form text field to enter their termination reason. So “insufficient patient accrual” could be described as “Study closed by PI due to lower than expected accrual” or “The study was stopped due to lack of enrollment”. So I used k-means clustering (after term frequency-inverse document frequency feature extraction) of the termination reasons to find groups of reasons that meant similar things, and then manually de-duped the groups (e.g. combining the “lack of enrollment” and “low accrual” groups into the same group because they meant the same thing).

I found that about 52% of terminated clinical trials end because of insufficient patient accrual. This implies that about 10% of clinical trials that end (either successfully, or because they’re terminated early) do so because they can’t recruit enough patients for the study.

termination_reasons

Predicting clinical trial termination?

Clinicaltrials.gov provides a bunch of information on each clinical trial–trial description, recruitment locations, eligibility criteria, phase, sponsor type (industry, institutional, other) to name a few–which begs the question: can this information be used to predict whether a trial will terminate early, specifically because of low patient? Are there visible aspects of a clinical trial that are related to a higher or lower probability that it fails to recruit enough patients? One might think that the complexity of trial eligibility criteria and the number of hospitals from which the trial can recruit from could be related to sufficient patient accrual.

Here was my attempt to get at a solution to this question analytically: fitting/training a logit regression multi class classifier–whether a trial would be “completed”, “terminated because of insufficient accrual”, or “terminated for other reasons”–on a random partition of clinical trial data, and measuring its accuracy at classifying out-of-sample clinical trials. The predictors were of two types: characteristic (e.g. phase, number of locations, sponsor type, etc.) and “textual”, or features extracted from text based data like the study’s description and eligibility criteria. Some of these features came from a similar tf-idf vectorization process as described in the k-means section above, other features were the simple character lengths of these text blocks. Below is a plot showing the relationship between two of these features: length of the eligibility criteria block of text, and length of the study’s title, two metrics that perhaps get at the complexity of a clinical trial.

complexity

The result: the logit model could only predict correctly whether trials would complete successfully, terminate because of low accrual, or terminate for other reasons 83.6% of the time. This is a pretty small improvement over saying “I think this trial will complete successfully” to every trial you come across, in which case you would be correct 80.6% of the time (see the Completed vs. Terminated pie chart above). Cancer clinical trials are very diverse, so it makes sense that there don’t seem to be any apparent one-size-fits-all solutions to improving patient accrual.

 

Standard
Knowledge

Amazon’s secret sauce: the flywheel model

Amazon’s flywheel of growth. From Andreessen Horowitz’s blog post http://a16z.com/2014/09/05/why-amazon-has-no-profits-and-why-it-works/

After finishing The Everything Store recently, I wanted to share an interesting framework that Bezos used when founding Amazon. The book, by the way, is a phenomenal read and gives great insight into Bezos’s character and how he has led an innovative Amazon. Ambition, persistence, spontaneity, and being neurotic/obsessive are some of the most common traits of the successful people I’ve read about so far, and he certainly embodies all of them.

Bezos thought about Amazon’s business model as a “flywheel” in the early days, and claimed that this was their secret sauce. Without going into what an actual flywheel is, this was another way of saying that the business model possessed a positive reinforcement loop that grew stronger if you fed any part of it. To quote the book:

… Bezos and his lieutenants sketched their own virtuous cycle, which they believed powered their business. It went something like this: lower prices led to more customer visits. More customers increased the volume of sales and attracted more commission-paying third-party sellers to the site. That allowed Amazon to get more out of fixed costs like the fulfillment centers and the servers needed to run the website. This greater efficiency then enabled it to lower prices further. Feed any part of this flywheel, they reasoned, and it should accelerate the loop.

Starting up the flywheel can be difficult, but once results accumulate, momentum builds and business accelerates. In the flywheel model, all incentives are aligned in the same direction. Some strategic and managerial conclusions:

  1. Design for success: the flywheel model is just another example of how leaders can design for the successful operation of their company before any real rubber hits the road. All planning and no action is bad, but having some sort of goal and a plan before doing any serious execution, in principle, works a lot more efficiently than trying things haphazardly and seeing what works.
  2. Design for alignment: a business model is least impeded when the result of anyone’s actions promote everyone’s desires and best interests, especially when that cycle is self-reinforcing.
  3. Do everything to start, protect, and build that initial momentum

Fun fact: during the 1999 holiday season, a lost box of stuffed Jigglypuffs wreaked havoc on a few Amazon distribution centers (they weren’t called fulfillment centers back then). Bezos ordered staff to pull all-nighters looking for the bundle of Pokemon–customers always came first.

Standard
Life

What trying to blog somewhat regularly does for me

A young Benjamin Franklin. I’m currently reading Isaacson’s biography of him: it’s brilliant, and Franklin was a baller. More to come later…

I have not been at all regular with my blog. I also have a bunch of draft posts on various topics just sitting there, partially written, mostly because I started writing and got stuck, or distracted, or ran out of time. Noticing that has made me want to write this, a short blog post about blogging (meta-blogging?).

Trying to blog somewhat regularly forces me to structure my thoughts, to come up with a cohesive, brief story that allows me to get my point across and hopefully get others thinking. This is something that I haven’t mastered yet–as evidenced by my collection of half-written draft posts–but I guess that’s the process of becoming a better writer, and where editing comes in. I wonder: do all the best bloggers edit their blog posts? Because I remember editing and revising essays for school over and over again, a process that took a lot of time. Some of the best bloggers that I follow seem to write off the cuff while maintaining brevity and an easy to follow structure in their posts.

The process of regularly structuring my point of view for writing also leads to the discovery of both holes in my thinking and also areas of opportunity that I can do more research on. Blogging also acts as a sort of accountability tactic: if I blog about doing something, then I feel even more compelled to do it. It certainly is a learning experience for me, and hopefully others can learn (about blogging, and about whatever else I talk about and share) along with me.

Standard
Life

Warren Buffett: insights into his character, obsession with OPM

Buffett’s house in Omaha, Nebraska. He bought it in 1957 and still lives in it today.

I tend to idolize Warren Buffett a little, something rekindled after recently reading Making of An American Capitalist.

He’s brilliant, humble, focused, self-confident, and frugal. He started his own “golf ball” business as a kid, employing an army of friends to fish out golf balls from ponds in local golf courses,  and then to clean, organize, and resell them. During his short time at Penn, Buffett joined a fraternity. He would spend parties at his frat house sitting on the ledge by a window, expounding on investing, the gold standard, and other economic concepts–a throng of guys and gals would always gather on the floor in front of him, hanging on his every word. In the early days of running his first fund, Buffett was insanely secretive about his investments, working from his home like a hermit, only wearing t-shirts and underwear, and refused to compromise on his fund’s 6 month lock-up period and $50,000 minimum investment (a lot at the time), even for celebrity investors. Those are just some of the captivating insights into Buffett’s character.

Buffett’s vast amount of wealth does not necessarily intrigue me that much–it is about how he build it: with self-reliance, focus, discipline, and authenticity.

He is also obsessed with “other people’s money”, or OPM, and OPM is essentially how he was able to build such a great fortune. One of Buffett’s first outright purchases of a company was an insurance company–he owns the well known GEICO today–and he used the float to fund his investments. That early purchase is said to be worth half of Berkshire Hathaway’s value today–this insightful post by Noh-Joon on Quora explains that, as well as how Buffett is able to essentially turn a 5% increase in actual investment appreciation into a 15% return (hint: leverage and effectively negative interest rates from insurance underwriting discipline). Not to mention, he’s a great stock picker.

Standard
Life

Some books I’m reading, and why

Ever since I got my Kindle 3 years ago, I’ve been reading more. A lot more: before my Kindle, I’d probably average less than one book a year (not including those required for school). For me, it seems that the convenience of reading was a big factor.

Why do I read? As Newton said, to “stand on the shoulder of giants”. So much of mankind’s history, so far, has been recorded in physical writing–not online. Books are still the best way to see into the minds of the greatest scientists, philosophers, leaders, businessmen, etc. of all time.

It’s a balance of course, between actually taking action, and sitting down to spend time learning. The two behaviors are not mutually exclusive: one does learn from taking action, usually skills. Experience–success and especially failure–can teach very important lessons. One can’t meet new people by reading books all the time either. Sometimes though, one can discover brand new ideas and ways of thinking by reading books, ideas that are only talked about in-depth through text, by those in our history who have made a big impact. It is a different way of broadening horizons and gaining perspective.

Onto a few of the favorite books that I am reading, or have read recently, and a short reason why:

  • Buffett: The Making of an American Capitalist, Roger Lowenstein
    • The first/original biography on Warren Buffett. Intriguing insight into who he was–and is–as a person, and what characteristics of his personality and events in his life made him so successful, walking the reader from early childhood through the rescue of the Salomon Brothers
  • Mindset, Carol Dweck
    • A book backed by lots of research studies on what the “growth mindset”  is, how it’s so related to success in life, and how to develop it. A good balance of theory and practicality.
  • One World Schoolhouse, Sal Khan
    • Khan, the founder of the successful and impactful Khan Academy, makes convincing arguments for school reform, and talks about his project and how Khan Academy is the start of an educational revolution. He also talks a little bit about his childhood and subsequent path to the founding of Khan Academy. Inspirational and informative.
  • Hooked, Nir Eyal
    • A very practical and impactful book on building habit forming products, products that have fiercely loyal customers who come back to use the product day after day, from someone who has lots of experience doing so. I’ve heard this one recommended a lot by my start-up friends, and it’s one that I often recommend as well, to entrepreneurs looking for a more practical, “business” book.
  • Power of Habit, Charles Duhigg
    • Basically, the science behind Hooked. I believe habits are one of the greatest “force multipliers” in life (definitely a post for another time), and in his book Duhigg presents the science behind them so we can better understand and utilize the power of habits.

You can see I prefer non-fiction, the reason why is pretty utilitarian.

Standard
Projects

How our talented team won $2500 at the TechCrunch Disrupt NYC Hackathon

corpsquare_screen1

We had an absolutely amazing and talented team at the TechCrunch Disrupt NYC 2014 Hackathon! Shout outs to our awesome front end designers Amanda Gobaud and Michelle Lee, and our tireless devs, Amine Tourki, Andrew Furman, and Teddy Ku. Here are the lessons that I learned from building a web application that won the $2500 Concur Technologies API first place prize.

  • Our app, CorpSquare (Concur + Foursquare), solved a problem. Several of the team members (me included) used Concur in the companies we worked for. So we had experience with problems or cool and practical use cases that an app designed around the Concur API could do. Even the  Concur VP of Platform Marketing told us afterwards that he had seen many with the problem we were trying to solve.
  • But, we also played the game strategically. Concur is a business expense tracking platform; most of their clients are big businesses. We felt that a business expense API wouldn’t seem as “exciting” or “sexy” as some of the other consumer-facing start-up APIs (Evernote, Weather Underground, to name a few). Since the different companies who sponsored the hackathon had API specific rewards for teams that used their API in the coolest way, this implied that there might be less competition for the Concur API reward. We made a “value” bet of sorts, as value investors would say–the strategy seems to have paid off.
  • Our team’s skills were complementarybut not too much so. A good hackathon team probably needs both design and dev skills, and different people should specialize in one or the other to make things most efficient. But, everyone should be well versed enough in non-specialty skills (like designers in dev, devs in design) to be able to communicate efficiently. For example, our designers were comfortable with both UI/UX design as well front end development like CSS. Several of our developers were full-stack, implementing the back end but also helping out with the front end. We used technologies (frameworks, languages) that we were all comfortable with, which, perhaps out of coincidence for us, was also an advantage.
  • Presentation matters, a lot. Our two wonderful front end designers spearheaded the movement to make our web application beautiful. With the help of everyone, beautiful it was. For the actual 60 second demo, we also selected the most energetic and enthusiastic speakers to present. First impressions matter, but when you’re being explicitly judged in comparison to at least 250 other people, and 60 seconds of talking and app visuals is all you’ve got, first impressions really matter.

Hindsight is 20/20, of course. Causally linking our tactics and strategies to our success is fuzzy at best. But learning never stops; whatever happens, success or failure, there is always something to take away and improve yourself, and others, with.

Standard