A picture's worth visual search: How image recognition will power analytics
•Download as PPTX, PDF•
4 likes•2,457 views
This document discusses the future of visual search and image recognition technologies. It begins with a brief history of visual search and outlines key areas like object, scene and logo identification. Google and Pinterest are highlighted as leaders in allowing visual searches of images. The document also explores emerging technologies like visual search capabilities from Slyce, Blippar and Houzz. It concludes by discussing potential applications of visual analytics and image data for audiences, brands and businesses.
Recommended
More Related Content
Viewers also liked
Viewers also liked (12)
Similar to A picture's worth visual search: How image recognition will power analytics
Similar to A picture's worth visual search: How image recognition will power analytics (20)
More from David Berkowitz
More from David Berkowitz (20)
Recently uploaded
Recently uploaded (20)
A picture's worth visual search: How image recognition will power analytics
- 1. A picture’s worth a thousand hashtags: How image recognition will power the future of analytics David Berkowitz Chief Strategy Officer Sysomos dberkowitz@sysomos.com www.sysomos.com @sysomos / @dberkowitz
- 2. About this presentation This talk was presented to Marketing Week Live in London in March 2017. A Texan version of this was delivered to W2O Group’s Pre-Commerce Summit during SXSW that month. If you prefer Frito pie to bangers and mash, I will gladly send you the W2O edition. Sources for information shown in slides are presented as links in the bottom- left corner. Further details about sources, where applicable, are also included in the notes field when downloading this presentation. To share your feedback or discuss this further, please contact me at dberkowitz@sysomos.com. Thank you. David
- 3. And now for something completely standard: an agenda • A brief history of nearly everything visual search • Why visual search matters • How Google, Pinterest, and others are deploying it • How marketers can use it • Numerous gratuitous references to all things British
- 5. Not British, but just a Chunnel ride away
- 6. How would you #HASHTAG the Mona Lisa?
- 7. #YOLO
- 8. Mona Lisa #Hashtags • #MonaLisa • #art • #painting • #woman • #lady • #smile • #smug • #Italian • #DaVinci • #epic • #outdoors • #masterpiece • #sky • #France • #LaGioconda • #badhairday • #beautiful • #Louvre • #Renaissance • #portrait This is a futile exercise. One can’t simply capture the Mona Lisa in hashtags. It points to the need for better ways to identify and analyze visual content. Text and hashtags alone don’t cut it.
- 10. Drop in a bucket visual The Visual Data GapRecall how pictures are worth a thousand words? There is so much more data in images (‘the sun’) than there is in text (represented metaphorically by the planets).
- 11. A Short History of Nearly Everything Visual
- 13. A challenge of Shakespearean proportions: “His reasons are as two grains of wheat hid in two bushels of chaff: you shall seek all day ere you find them, and when you have them, they are not worth the search.” -Bassanio, The Merchant of Venice
- 14. Number of object categories out there 15,000There are 15,000object categories Source: IEEE
- 15. Source: Computer Vision by Richard Szeliski For fun, I included a few examples of early attempts at machine- powered object recognition.
- 16. Source: Computer Vision by Richard Szeliski
- 17. An inflection point waiting to happen Jason Goldberg, Razorfish: “I’m strongly bullish on visual search. It solves a real problem consumers have… In the not- too-distant future, it’ll become a heavily used mainstream feature. I think the inflection point is at least a year away, but not two years." We’re approaching the inflection point, but it has taken longer than expected. This report is from November 2014.
- 19. This is simply an enlarged, cropped version of the highlights from the previous chart.
- 20. You can’t always get what you want (with text search) • 74% of consumers say text-based keyword searches are inefficient for helping them find the right products online • 67% of consumers say quality of product images is very important in selecting and purchasing products • 90% of information transmitted to the brain is visual • Visual information is processed 60,000x faster than text Source: Slyce
- 21. The dress that inspired Google Image Search in 2001
- 22. The dress that inspired Google Image Search in 2001 “…People wanted more than just text. This first became apparent after the 2000 Grammy Awards, where Jennifer Lopez wore a green dress that, well, caught the world’s attention. At the time, it was the most popular search query we had ever seen. But we had no surefire way of getting users exactly what they wanted: JLo wearing that dress. Google Image Search was born.” -Eric Schmidt, Executive Chairman, Google Source: Project Syndicate
- 23. Solving the Clarissa problem My wife gave me this reference. As a kid, she always wanted to identify and shop for whatever Clarissa wore.
- 24. Applications for image recognition Source: Facebook
- 26. An introductory framework for visual search
- 27. Layers of image recognition A Deep Learning algorithm is presented with the images made up of simple pixels. The algorithm discovers simple regularities that are present across many/all images, like curves & lines. The algorithm discovers how these regularities are related to form higher-level concepts The system gains a high-level understanding of the original image… all automatically Source: GazeMetrix
- 28. A framework for visual search Scene Identification Intelligence Object Identification Intelligence Logo Identification Intelligence Image Identification Intelligence Category Identification Intelligence This notes some of the most important processes within visual search. Also note that identification and intelligence are two separate approaches. Examples follow.
- 29. What follows is an example using a real photo from Agnes, a Chinese tourist to the UK.
- 30. Here’s her photo. In each subsequent slide, you can see how the framework plays out and a sample finding that can be derived. Note the intelligence examples that follow are for illustrative purposes only; feel free to cite the framework, but not the data itself.
- 31. Category Identification: This is food and drink Category Intelligence: 7.2% of images posted at museums include food or drinks
- 32. Logo Identification: There is a Fanta logo, and the text in the top-right says Starbucks Logo Intelligence: Fanta logos are rarely paired with Starbucks; Fanta logos are most often seen with Coca-Cola and Adidas
- 33. Object Identification: This looks like fish and chips with a can of Fanta Lemon Object Intelligence: Fanta Lemon is the fourth most popular soda when paired with fish & chips
- 34. Image Identification: This is the same photo that appears on Agnes_Cin’s Flickr and public Facebook pages Image Intelligence: This image hasn’t been shared in any media outlets and hasn’t been shared publicly
- 35. Scene Identification: This photo seems to be taken outdoors during the day Scene Intelligence: 94% of photos at the British Museum are shot indoors, compared to 87% of museum photos worldwide
- 37. This section is drawn (no pun intended) from quickdraw.withgoogle.com.
- 43. Pinterest: one of the world’s biggest search engines • 150 million monthly users • 75 billion pins • 2 billion searches/month • 97% of searches are unbranded Source: Pinterest
- 44. Pinterest: search pins from real-world images Source: Pinterest Examples from Pinterest’s new Visual Discovery follow. In the downloadable version of this talk, the next few visuals play as GIFs, and you can read more at the source below.
- 45. Pinterest Visual Discovery Source: Pinterest
- 46. Browse images and buy from them Source: Pinterest
- 51. I really liked the circled image here: Pinterest draws a connection between Big Ben and the Blue Mosque. Such errors are often more revealing.
- 53. Toys R Us offers Slyce image detection for its catalog Source: Slyce
- 54. Visual search to complement textual search eMarketer: Do you think [visual search] will replace some types of searches, or do you think it will augment existing searches? Gierhart: It will probably augment. It’s adding a new utility to what was there before... There will still be contexts for both. Source: eMarketer (see a related video on YouTube)
- 55. Blippar: scan images for surprises about Planet Earth Source: Blippar
- 56. Source: Houzz Houzz has a Visual Match offering akin to Pinterest.
- 57. TheTake uses AI to identify products, locations in video Source: TheTake
- 58. What’s Possible with Image Analytics The images that follow are sample reports drawn from Sysomos. The data is again for illustrative purposes. Reach out if you want to dive deeper into any of this.
- 59. Visual analysis: understanding visual characteristics Logo recognition Object recognition Scene recognition Food recognition Color detection OCR: Search text within images
- 60. Visual analysis: understanding visual characteristics Logo recognition Object recognition Scene recognition Food recognition Color detection OCR: Search text within images
- 61. Visual analysis: understanding visual characteristics Logo recognition Object recognition Scene recognition Food recognition Color detection OCR: Search text within images
- 62. Audience analytics show growth and spikes
- 63. Brand affinities can highlight cross-promotion opportunities
- 64. Identify most popular objects, scenes
- 65. B2B applications for visual search Creative optimization Influencer marketing Rights management Crisis management Partnership ideation Competitive intelligence Customer service
- 66. Any questions?
- 67. So long, and thanks for all the fish. Let’s take tea! David Berkowitz Chief Strategy Officer Sysomos dberkowitz@sysomos.com www.sysomos.com @sysomos / @dberkowitz
Editor's Notes
- A picture’s worth a thousand hashtags: How image recognition will power the future of analytics • Understand how artificial intelligence, the surge of visual content creation, and other trends lead to visual search • Discover what you can find by searching images that doesn’t show up in text search • Explore the kinds of insights that you can unearth from those visual search results • See how visual search can then fit in with your overall marketing plan, from content marketing to customer service
- http://www.w2ogroup.com/highlights-w2o-groups-2017-precommerce-summit-pt-2/
- First, a question…
- This is where we’ll focus today
- The amount of data available from visuals versus text is like comparing the Earth to the sun
- A picture’s worth a thousand hashtags: How image recognition will power the future of analytics • Understand how artificial intelligence, the surge of visual content creation, and other trends lead to visual search • Discover what you can find by searching images that doesn’t show up in text search • Explore the kinds of insights that you can unearth from those visual search results • See how visual search can then fit in with your overall marketing plan, from content marketing to customer service
- And now, some history… https://en.wikipedia.org/wiki/Bill_Bryson
- The challenge with image search – courtesy of William shakespeare http://www.rhymezone.com/r/gwic.cgi?Path=shakespeare/comedies/merchantofvenice/i_i//&Word=have+them,+they+are+not+worth+the+search.#w His reasons are as two grains of wheat hid in two bushels of chaff: you shall seek all day ere you find them, and when you have them, they are not worth the search. http://www.rhymezone.com/r/gwic.cgi?Path=shakespeare/comedies/merchantofvenice/i_i//&Word=have+them,+they+are+not+worth+the+search.#w Image: https://www.biography.com/.image/c_fill,cs_srgb,dpr_1.0,g_face,h_300,q_80,w_300/MTE1ODA0OTcxNzgzMzkwNzMz/william-shakespeare-194895-1-402.jpg
- http://ieeexplore.ieee.org/document/7298599/
- Computer Vision: Algorithms and Applications Richard Szeliski Springer Science & Business Media, Sep 30, 2010 - Computers - 812 pages https://books.google.com/books?id=bXzAlkODwa8C https://books.google.com/books?id=bXzAlkODwa8C&printsec=frontcover&source=gbs_ge_summary_r&cad=0#v=onepage&q&f=true
- A lot goes into this whole image recognition thing Computer Vision: Algorithms and Applications Richard Szeliski Springer Science & Business Media, Sep 30, 2010 - Computers - 812 pages https://books.google.com/books?id=bXzAlkODwa8C
- http://blog.slyce.it/wp-content/uploads/2015/11/Visual_Search_Technology_and_Market.pdf Note Rolling Stones reference (British) Source: Slyce, summer 2015 After last bullet: shouldn’t this whole talk be in emoji?
- http://www.vogue.co.uk/article/j-lo-green-versace-dress-responsible-for-google-image-search “In an interview charting the evolution of the internet-service provider, Schmidt revealed that following the singer's appearance on the red carpet at the 2000 Grammy Awards wearing that diaphanous, slashed-to-the-navel Versace dress, the relatively new Google search engine was inundated with people wanting to see it - except that they couldn't.” ---- So our co-founders Larry Page and Sergey Brin – like all other successful inventors – kept iterating. They started with images. After all, people wanted more than just text. This first became apparent after the 2000 Grammy Awards, where Jennifer Lopez wore a green dress that, well, caught the world’s attention. At the time, it was the most popular search query we had ever seen. But we had no surefire way of getting users exactly what they wanted: JLo wearing that dress. Google Image Search was born.
- http://www.vogue.co.uk/article/j-lo-green-versace-dress-responsible-for-google-image-search “In an interview charting the evolution of the internet-service provider, Schmidt revealed that following the singer's appearance on the red carpet at the 2000 Grammy Awards wearing that diaphanous, slashed-to-the-navel Versace dress, the relatively new Google search engine was inundated with people wanting to see it - except that they couldn't.” ---- So our co-founders Larry Page and Sergey Brin – like all other successful inventors – kept iterating. They started with images. After all, people wanted more than just text. This first became apparent after the 2000 Grammy Awards, where Jennifer Lopez wore a green dress that, well, caught the world’s attention. At the time, it was the most popular search query we had ever seen. But we had no surefire way of getting users exactly what they wanted: JLo wearing that dress. Google Image Search was born.
- http://ib.huluim.com/show_key_art/20872?size=1600x600®ion=US
- https://code.facebook.com/posts/561187904071636/segmenting-and-refining-images-with-sharpmask/
- It’s an exciting time – Facebook is literally giving its technology away http://www.theverge.com/2016/8/25/12630850/facebook-fair-deepmask-sharpmask-ai-image-recognition
- A picture’s worth a thousand hashtags: How image recognition will power the future of analytics • Understand how artificial intelligence, the surge of visual content creation, and other trends lead to visual search • Discover what you can find by searching images that doesn’t show up in text search • Explore the kinds of insights that you can unearth from those visual search results • See how visual search can then fit in with your overall marketing plan, from content marketing to customer service
- https://www.gazemetrix.com/
- Visual Search Framework Category identification: that is a green dress Image identification: that is a photo of Jennifer Lopez wearing a green dress from the 2000 Grammy Awards Object identification: that is a Versace green dress, and here it is offered by various retailers and shown on people other than J-Lo Category intelligence: green is the fifth most popular color of dresses this year Image identification: that image was shared by 29,400 people and media outlets Object identification: Object intelligence Scene identification: those are trees and mountains and it’s daytime and there are seven people all in this photo Scene intelligence: 29% of photos of that green dress are shown in daytime, and 8% are paired with luxury handbag brands
- Logo can apply to text as well
- A picture’s worth a thousand hashtags: How image recognition will power the future of analytics • Understand how artificial intelligence, the surge of visual content creation, and other trends lead to visual search • Discover what you can find by searching images that doesn’t show up in text search • Explore the kinds of insights that you can unearth from those visual search results • See how visual search can then fit in with your overall marketing plan, from content marketing to customer service
- https://quickdraw.withgoogle.com/ http://www.theverge.com/2016/11/15/13641876/google-ai-experiments-quick-draw-image-recognition-game
- A picture’s worth a thousand hashtags: How image recognition will power the future of analytics • Understand how artificial intelligence, the surge of visual content creation, and other trends lead to visual search • Discover what you can find by searching images that doesn’t show up in text search • Explore the kinds of insights that you can unearth from those visual search results • See how visual search can then fit in with your overall marketing plan, from content marketing to customer service
- https://business.pinterest.com/en/blog/introducing-search-ads-pinterest
- https://blog.pinterest.com/en/search-outside-box-new-pinterest-visual-discovery-tools
- https://blog.pinterest.com/en/search-outside-box-new-pinterest-visual-discovery-tools
- When you’re ready to buy, you can check out right on Pinterest, or get things directly from your favorite brands. CB2, Macy's, Neiman Marcus, Target, Wayfair and other brands will all be offering shoppable looks, thanks to Pinterest’s new partnerships with Curalate, Olapic, Project September, Refinery29 and ShopStyle.
- Bangers and Mash – Pinterest Visual Discovery
- Prince George Pinterest Visual Search
- Prince George Pinterest Visual Search
- Big Ben Pinterest Visual Search
- Big Ben Pinterest Visual Search
- A picture’s worth a thousand hashtags: How image recognition will power the future of analytics • Understand how artificial intelligence, the surge of visual content creation, and other trends lead to visual search • Discover what you can find by searching images that doesn’t show up in text search • Explore the kinds of insights that you can unearth from those visual search results • See how visual search can then fit in with your overall marketing plan, from content marketing to customer service
- http://blog.slyce.it/2016/10/26/toysrus-great-big-holiday-book/
- eMarketer: Do you think [visual search] will replace some types of searches, or do you think it will augment existing searches? Gierhart: It will probably augment. It’s adding a new utility to what was there before. There are probably several situations where you still are not going to be able to find that thing you’re thinking of, so you’re going to want to search. There will still be contexts for both. eMarketer: What prompted you to decide to pull the trigger on visual search now? Gierhart: We wanted to be first to market. I think visual search is the next thing that everyone’s going to start rolling out in the next year. So we just wanted to get out there and start trying it with our customers so we could figure out how to improve it. - See more at: https://www.emarketer.com/Article/CMO-One-to-One-Neiman-Marcus-Readies-Visual-Search-Shoes-Handbags/1011580#sthash.ym76vUpl.dpuf
- https://blippar.com/en/showroom/#planet-earth-ii Get closer to the natural world with Blippar and Planet Earth. Scan one of seventeen everyday objects to reveal one of the incredible animals featured in the series. See the creature up close and learn some surprising facts. Try it yourself now by scanning your hand, a rose or a fork. Selected regions only. www.blippar.com
- Houzz Visual Match http://blog.houzz.com/post/150412447953/houzz-introduces-visual-match-making-it-even eMarketer interview with Liza Hausman http://totalaccess.emarketer.com/Interview.aspx?R=6002271&dsNav=Ntk:relevance%7cliza+hausman%7c1%7c,Ro:0,Nr:NOT(Type%3aComparative+Estimate) https://www.emarketer.com/Interview/Use-of-Augmented-Reality-Image-Search-Boost-Likelihood-of-Online-Home-Product-Purchases/6002271 eMarketer: You recently introduced another tool called Visual Match. What is it? Hausman: Think about Visual Match as an automatic search. For any image you see on Houzz, it’s automatically looking for matches and will start recommending similar photos. eMarketer: What has been the reaction to Visual Match? Hausman: It’s been phenomenal. One of the early popular features on the Houzz app was these green tags that professionals and brands could add to their photos, to identify the products and materials in the photos. Houzz users loved the ability to not only see a photo and get excited about it, but also find out what’s in the photo and where to get it. A lot of our development lately is focused on making that easier. Both View in My Room and Visual Match came out of this desire—they help people not just discover what they love, but take the next step and transact.
- https://techcrunch.com/2017/02/13/working-with-major-studios-thetake-launches-ai-image-recognition-engine-for-businesses/amp/ Focus is now a B2B model to partner with studios, networks https://thetake.com/products?mId=2791&mName=Fifty%20Shades%20Darker&mImage=https%3A%2F%2Fimg.thetake.com%2Fmovie_images%2F9c15cd823da4514f36ca1ac80cbdcb86df39d8b52ca16d813ffb2d02061ea8fc.jpeg
- A picture’s worth a thousand hashtags: How image recognition will power the future of analytics • Understand how artificial intelligence, the surge of visual content creation, and other trends lead to visual search • Discover what you can find by searching images that doesn’t show up in text search • Explore the kinds of insights that you can unearth from those visual search results • See how visual search can then fit in with your overall marketing plan, from content marketing to customer service
- Multi select of Objects / Scenes/ Food etc. Understanding visual characteristics TASK: Find me all photos with “Coca Cola logo” taken at a beach with a ball OR balloon in the picture
- Task: Find me all photos with “Coca Cola logo” taken at a beach with a ball OR balloon in the picture Task: How often do people drink Coca Cola with “Burgers” vs. “Pizza” in Europe Task: Most popular ‘colors’ of Coke ‘cans’ at different ‘scenes’
- Task: Upload a list of user segments and understand their conversations, preferences and behavior patterns
- Task: Identify partnership opportunities – co-branding and join promotion
- Task: More intuitive and granular reporting along with geographic and behavior insights.
- Creative optimization Influencer marketing Rights management Leak prevention Creative – Buster of John Lewis: https://s3-eu-central-1.amazonaws.com/centaur-wp/creativereview/prod/content/uploads/2016/11/Buster-The-Boxer-John-Lewis-Christmas-Ad-07.jpg Influencer: Emma Willis of The Voice, Big brother Rights management – Saul Goodman – Better Call Saul - http://static.tvtome.com/images/genie_images/news_hub/uploaded/TimSpotnews137893011780/breakingbad_saul.jpg Help image: http://images6.fanpop.com/image/photos/36600000/The-Beatles-image-the-beatles-36630507-500-422.jpg Partnership ideation image: Sherlock / Watson - http://i.dailymail.co.uk/i/pix/2014/07/28/article-2708469-0C0AAB1D000005DC-912_634x395.jpg Competitive intelligence: Jessica Ennis-Hill, Olympian - https://static.independent.co.uk/s3fs-public/styles/story_medium/public/thumbnails/image/2015/12/05/09/Jessica-Ennis-Hill.jpg Customer service – British Airways flight attendant - http://i4.getwestlondon.co.uk/incoming/article9033228.ece/ALTERNATES/s615/JS60931715.jpg
- Summary slide (didn’t use) Technology progress around image recognition is moving faster than consumer behavior is shifting There are many interlocking components to visual search, and progress is happening everywhere We mammals are better adapted to visual than textual search There will always be a gap between what’s shared textually and what’s shared visually The J-Lo/Clarissa problems have been practically solved; marketers must prioritize consumer education There are many applications for visual search data: research, customer service, influencer marketing, creative optimization crisis management, and many more
- A picture’s worth a thousand hashtags: How image recognition will power the future of analytics • Understand how artificial intelligence, the surge of visual content creation, and other trends lead to visual search • Discover what you can find by searching images that doesn’t show up in text search • Explore the kinds of insights that you can unearth from those visual search results • See how visual search can then fit in with your overall marketing plan, from content marketing to customer service