Apple Vision Pro – What About Comic Books?

Obviously the Vision Pro isn’t the first VR/AR/whatever device, but it’s the first one that’s got enough resolution to prevent me from getting pissed off as soon as I stick it to my face.

And while there have already been reams of articles published about other aspects of using the Vision Pro, I haven’t seen anybody yet discuss what it’s like to read a comic book with it. So… here you go.

To set the stage, I’ll have to say that I’ve never really loved reading comics on my laptop or iPad. It’s mostly fine, but it’s not the same as what you’ve got after your Wednesday comics run.

See, comics artists (writers and illustrators both) still do think about how the story is presented in a traditional paper format. And many times that includes how two pages, side-by-side, will tell the story. Sometimes that’s a two-page spread, sometimes it’s just knowing how the reader’s eyes will flow as they’re looking at the individual panels across those two pages.

If you’re reading on a tablet or a laptop, that two-page experience is compromised relative to a ‘real’ comic book. You either read a page at a time (on a vertically oriented iPad for example) in order to see it at a nice size, or else you put up the two pages on the screen at the same time and then they get… small.

Because unless you’ve got a display that’s at least 13 inches wide and a little more than 10 inches tall, you’re tinier than a standard comic. There’s a few monster laptops that get close, but I’ll never buy such a beast. And I’m also not going to go sit at my desk with the big monitor to read a comic book – comics are a laying-on-the-couch (or in bed) activity.

Enter Vision Pro. There are no limits to the page size because the screen is filling your field of view.

So far I’ve only tried out Apple’s ‘Books’ app and it’s got a few issues but overall it’s usable. To be clear, this isn’t yet a Vision Pro specific app – I’m just using the iPad version.

(And for some reason Volume 1 of the Sandman series was only $3.99 so that was a nice quick purchase for purposes of testing, even though I already own several other versions of it.)

To use, just open the Books app, drag it to a comfortable size and… boom. It’s a big version of the comic hovering in space in front of you. But the interaction is, at this point, really fiddly. Page turns are apparently supposed to work by touching your thumb to finger and then dragging but I would often experience lag and there were times where the page actually turned back instead of forward. And it kept getting into a mode where it would highlight a full page and then inexplicably give me a prompt for ‘copy’

I assume these issues will be worked out once there’s a native Books app, not to mention some of the other comics reading solutions out there that will undoubtedly show up soon. So I’m not going to dwell too much on problems I’m experiencing on day one.

But I will say that visually, it’s really really nice. It’s up to you to choose where you want to place the book in space (i.e. how far away it feels) as well as how large it is. For my first attempts I mostly ended up with something that felt like it was about 2 feet wide, positioned about the same distance in front of me.

But I was also curious about zooming wayyy in, particulary if the source material supports it. So I did a quick scan of my own to test. And I’ve gotta say, it’s pretty fun to have something like this floating in front of you…

and then walk up and/or zoom until you can see this:

or even this:

(Yeah I know, this isn’t strictly speaking something you need a Vison Pro to do… but it’s a different feeling with grape-sized halftone dots within arm’s reach vs. just zoomed up on a flat display).

I also wondered about alternate methods of displaying things. Although the Books reader is just a planar 2-dimensional window, Apple does support an image-viewing mode for panoramic images that makes it very large and then curves the image around you.

So, just for grins, I went ahead and stitched a few comics pages side-by-side (including a double-page spread) and tossed it up to see how it would feel.

And I like this experience a lot, actually. The photo at the top of this post gives you a rough idea for what it’s like, but obviously you need to experience it directly. It’s extremely immersive… like you’re within the book instead of just looking at a page or two. I couldn’t find a way to adjust how big and how much curvature is in this mode – the default was probably more than I wanted really.

But it’s pretty neat to just look a little to your left if you want to go back a page or two to remind yourself of something you just read. On the other hand, you could argue that this scenario sort of derails the artist’s intended experience since you can now peripherally see the next two pages before you get to them…

Anyway, I’m sure there will be artists and developers experimenting with how best to view and interact with existing comics using this thing. And beyond that, there will also be new comics that take advantage of this unbounded space in new ways. Individual panels floating in space over massively detailed backgrounds, oddball aspect ratios, etc.

(I vaguely remember some comic (maybe by Mike Allred?) that opened ‘upwards’ vertically (i.e. stapled on the top) to get a really tall page. Hang on… googling… Ahhhh yes. The comic was actually called ‘Vertical’ and written by Steven T. Seagle with art by Allred. Published by Vertigo in 2004. Cool. I’ll have to go read that again with the top of the page towering over me.)

And someone go turn J.H. Williams III loose on this new boundary-less paradigm, because if there’s anybody who wants to break free of the page, it’s him.

Bottom line – I know I’ll always enjoy the tactile pleasure of actual paper in my hand, but I think this new spatial view is going to be a VERY fun alternative. If anybody’s got further questions about the comic-reading experience on this device (or has experience with other VR devices that they can share), please comment below!

ADDENDUM: Check out the comment below from Rotem Cohen, who has created a Vision Pro comic book reader that includes some of the stuff I talked about above. It’s really nicely done!

Camera Viruses!!! OMG!!! (Fear-mongering)

Now that we’re finally starting to see some mainstream cameras that can run apps (see previous discussion here) I suppose it’s time to talk a little bit about one of the potential downsides of such a scenario.

Because as soon as you allow people to download and run third-party software, you’ve got to start thinking about what viruses might come along with those apps. (And I’m using ‘virus’ generically, including any sort of malicious code that might run without your knowledge on the camera).

The first thing that comes to mind are the privacy concerns – because if your camera also features network connectivity that means someone could surreptitiously redirect your photos to a remote server somewhere. Bad enough if you’re being monitored by a spouse or an employer, but there may be even more consequential cameras to infiltrate…

(Keep in mind too that cameras are ALL going to have GPS chips embedded in them soon enough… instant tracking/surveillance device! Even if you’ve got the lens-cap on there’s probably a microphone on your camera too… one that can potentially be remotely activated and then monitored)

Most viruses are of course created for financial gain. Infect a remote computer and then use it as a server for sending out prescription-drug or junk-enhancement advertisements. What’s the photo equivalent?

If we consider the increasingly powerful processors that are coupled with cameras these days – processors that will eventually be able to run real-time photorealistic rendering algorithms – it’s not unreasonable to expect a virus that is capable of actually altering stills or videos while they’re still in camera.

True, that could just be a bit of fun – the so-called Walter White virus, which replaces everybody’s face with Brian Cranston’s:

(Been catching up on Breaking Bad lately…)

But (back to the financial incentives) what about a virus that keeps an eye your photos in order to identify branded objects – someone holding a can of Heineken, someone wearing Nikes, etc. Beyond just providing market research on your personal buying habits, a sophisticated enough algorithm could even REPLACE some of those logos with competing products. Not only does this hijack the unstated endorsement of a brand when you post one of those images to Facebook, it effectively is also altering your memories of the event. Your personal history purchased by the highest ethics-challenged bidder.

What else? Photo forensics (altering the date/time/location of a photo is easy enough, but it also makes it MUCH more difficult to identify photo-retouching if it occurs BEFORE an image is converted to a JPEG and stored to memory).

Financial transactions. My bank allows me to take a photo of a check in order to deposit it. What if that image gets altered before I send it to the bank…???

Security cameras. The movie trope of replacing security camera footage with alternate footage will be more and more of a concern.

Sporting events. There’s a lot of financial incentive to being able to affect the outcome of a major game. Hack the instant-replay camera – the one they go to when reviewing a play?
Any other ideas? What’s your best (or most outlandish) guess for the sort of camera viruses we might see out in the wild?

Mirror Box

Bored?

Buy a box of 6 mirror tiles ( i.e. something like this )

Use some duct-tape and make a box out of them (mirror-side inwards)

(put a little duct-tape handle on the 6th tile to make it easy to take it on and off)

Set timer on camera, (turn flash on), place camera in box, put mirror ‘lid’ onto box before shutter fires…

…get cool photo.

toss in balls of aluminum foil – because they’re shiny.

Play with light-leaks at the seams of the box.

Add candles.  Pretty.

Finally, add Droids.  Everything is better with Droids.

(Click on any of the above images to biggerize them.  Also on Flickr, including a few more.)

 

UPDATE:  Here’s someone who got inspired about the idea and did a MirrorBox photo-a-day of throughout January 2012  http://flic.kr/s/aHsjxK8ZKe  Cool!

Beyond Focus (Part 1)

A lot of buzz lately on Lytro, the company who’s building a new camera based on lightfield (aka plenoptic) technology.  In a nutshell, they’re building a camera that will allow you to refocus your photos after you’ve taken them.  (Go poke around here if you want to see some examples of what is being discussed.)

And it is indeed some very cool technology (although I’m a little surprised that it’s taken this long to productize it given the fact that the basic concepts have been floating around for a long time now).  (Okay, technically they’ve been floating around for a really, really long time now).

But I think it’s worth talking about how this fits into the workflow of a typical photographer.   Because, although there’s been no shortage of people willing to label this as a ‘revolution’ in the way we do photography, I personally see this as more of a feature – one more thing that will be nice to have in your bag of tricks.

A useful trick, to be sure, but if you lined up all the features I’d like in a camera it would probably be somewhere in the middle of the pack.  Somewhere (far) below a high resolution/high dynamic range sensor.  Somewhere below good ergonomics.  Maybe (if it’s well-implemented) I’d choose this over an extremely high burst-speed or an integrated Wi-Fi connection.  Maybe.

But it’s the ‘well implemented’ part that worries me.  Because it’s not a feature that would compel me to buy a particular camera above all others.  There are always trade-offs when you’re creating a camera and the Lytro camera will be no exception.  So unless they manage to execute extraordinarily well on all the other aspects of their camera, it could easily turn into something that’s just not all that compelling for most people.  If I said there is a camera out there that can capture 100 megapixel images you might think that’s a very cool device, right?  But then if you find out that it can only shoot at ISO 50 – that it produces absolutely unusable images unless you’re shooting in bright sunlight… Well, that camera is suddenly a lot less interesting.

So, in much the same way as what we’ve seen with the Foveon sensor – a cool technology that was never paired with a camera which was all that good in other areas – we’ll just have to wait and see what the eventual product from Lytro will look like.  I’m optimistic – there’s some smart folks there – but it’s not a trivial challenge.

I think the best thing we can hope for is that this sort of feature doesn’t remain something that is only available from Lytro.   Hopefully we’ll see similar capabilities coming from the other manufacturers as well.  Yes, there are definitely patents in place on a lot of this but there are also a number of different ways to implement light-field image capture.  For instance, you can (as I believe is the case with Lytro) use a lens that captures multiple viewpoints and sends them all to a single large sensor.  Or, you can use multiple lenses each coupled with their own sensors.  Either way you get an image (or set of images) that will require some post-processing to produce something that’s human-friendly.

Which is fine.  Something I’ve been predicting for years (and have said so many times on the This Week in Photography podcast) is that the most common camera we’ll see in the future will be a computational camera.  And personally I’m willing to bet that more often than not it will be composed of an array of smaller lenses and sensors rather than a single monolithic lens/sensor.

Why?  Big sensors are expensive to manufacture, largely due to the fact that a single defect will require you to scrap the whole thing.  Big glass is also expensive – it’s very difficult to create a large lens without introducing all sorts of optical aberrations.  (Although smart post-processing will be increasingly important in removing those too).

So it’s clear to me that there’s a better and more cost-effective solution waiting to be developed that uses an overlapping array of inexpensive cameras.  (Lytro started out this way, incidentally, via the Stanford Multi-Camera Array)

The camera in your phone, or the one stuck in the bezel of your laptop, costs only a few bucks when purchased in quantity.   Now make yourself a 5×5 array of those things.  If each one is 8 megapixels (easily available these days), that’s a lot of resolution to play with.  No, you won’t be able to generate a single 200Megapixel image but you’ll still be able to get some serious high-rez imagery that also has a lot of additional benefits (see my next post for more on these benefits).

And yes, I’m completely ignoring a ton of engineering challenges here, but there’s nothing I’m aware of that feels like it can’t be solved within the next few years.

Bottom line:  Don’t be at all surprised if the back of a future iPhone looks something like the image at the top of this post.  I’m not gonna guarantee it, but I’m willing to take bets…

(Click here for Part 2 of this discussion)

Beyond Focus (Part 2)

So let’s talk some more about the ramifications of the lightfield/plenoptic technology that I looked at in my last post.  For Lytro, the marketing push (and the accompanying press) are all about the ability to re-focus the image after the photo’s been taken, as a post-process.   And they show you these massive foreground-to-background refocuses.  But that’s really just a parlor trick – how often are you suddenly going to decide that the photo you shot would be much better off if the background were in focus instead of the person standing in the foreground?

On the other hand, being able to do a very subtle refocus – for example to make sure that the eyes in a close-up portrait are perfectly sharp – that has real value to almost all photographers and there’s many a shot I’ve taken where I wish I could do exactly that!

But there’s actually a lot more to this technology than just refocusing. In reality what you’ve got here (or more specifically what can be computed here) is an additional piece of data for every pixel in an image – information about the depth or distance-from-camera.

So it’s not just the ability to refocus on a certain object in the image, it’s an overall control over the focus of every depth plane.  The narrow-focus ‘Tilt-Shift’ effect becomes easy.  You can even have multiple planes of focus.  And macro photography is almost certainly going to see a big benefit as well.

While we’re at it, you’ll also have the ability to choose the exact characteristics of the out-of-focus areas – the bokeh.  This would include the ability to create ‘stunt bokeh’ similar to what certain lensbaby  products can produce (see here).

Oh, and it’s also pretty easy to generate a stereo image pair, if you’re into that sort of thing…

But wait, there’s more!  Making use of depth information is something we do all the time in the visual effects world.  Consider the image below.

Here’s the depth image for that scene, where brighter areas are (obviously) closer to camera.

In the same way that we can use this information to choose where to focus, we can apply other image adjustments to different depth areas.  Want to introduce some atmosphere?  Just color-correct the ‘distant’ pixels to be lower contrast.

From there it’s just a step away to compositing new elements into the scene – anything from a snowstorm to ravenous rampaging raptor robots.

Bottom line?  Computational photography in its many forms is the future of photography.  We’re almost certainly going to see the single-lens, single-sensor paradigm go away and we’re absolutely going to live in a world where more and more of the image-creation process occurs long after you’ve actually ‘taken’ the photograph.   Personally, I’m looking forward to it!

Shades of Gray

There are two types of people in the world – those that divide things into two categories and those that don’t. I’m one of those that don’t :-)

Okay, look… I get it. Our brains are pattern-finding devices and nothing’s easier than putting things into either/or categories. But not everything is easily categorizable as Right or Wrong, Good or Evil, Black or White… in fact virtually nothing is. It’s a mixed-up muddled-up shook-up world, right? So when I get into discussions with people who see nothing but absolutes, I tend to push back a bit. At least in my opinion, that sort of Dualistic worldview is not just lazy thinking, it can quickly grow dangerous if it prevents people from considering all aspects of a situation.

The particular conversation that sparked this blog post somehow turned to the Taijitu – the classic Yin/Yang symbol – which has for a lot of people has apparently come to embody this black-and-white worldview. Disregarding the fact that the Taijitu is a lot more nuanced that that, and is more about balance than absolutes, I decided to see if I could come up with something that more explicitly acknowledges the shades of gray that exist in the real world. A bit of imageprocessing mojo later, and I had this:

From a distance it maintains the general appearance and symmetry of the classic yin/yang ratio but up close we see the real story – the edges are ill-defined and chaotic, nowhere is the symmetry perfect, and most of it is composed of shades of gray.

I’m not sure what exactly to do with this now, but just in case anybody thinks it’s cool I’ve stuck a high-resolution version of it up on flickr with creative common licensing. Do with it what you will, and if someone wants to put it on a T-shirt or something, could you send me one? :-)

iPhone: The Perfect Camera?

I’ve been saying for a long time now that cameras need to evolve to where they’re an open computing platform.  To where all of the hardware on the device can be programmatically controlled by onboard software applications. Unfortunately we haven’t seen a whole lot of movement in this area from the big camera manufacturers, other than a bit of SDK support by Canon and (finally) Nikon and some interesting but cumbersome hackish options.  

I know that part of the reason for this is that the software/firmware on a camera isn’t really designed with this in mind – it’s not necessarily an easy change to develop an architecture that would support 3rd party applications.  Which is why I’m starting to think that this will end up being solved in the other direction – by dedicated computing platforms that also happen to have camera capabilities.  Platforms like, for instance, the iPhone.

Now clearly the title of this post is intended to be a bit ridiculous.  I’ll be the first in line to talk about how crap the current iPhone camera is.  But the limitations are primarily due to the hardware.  And camera hardware, at least up to a certain point, is a really cheap commodity.  

So let’s talk a little bit about what I could do if I had a device with decent camera hardware (a reasonably good sensor, onboard flash, maybe some other worthwhile stuff like GPS, WiFi, etc.) along with full access to that hardware via a real programming interface like the one available for the iPhone.  Here’s just a few ideas:

-Timelapse.  This is such a simple feature from a software perspective yet many (probably most) cameras don’t support it.  For my DSLR I need to buy an external trigger with timer capabilities.  Why?  (Other than the obvious answer, which is because it allows the camera manufacturers to charge a ridiculous amount of money for such a remote trigger.  Hint – buy an ebay knockoff instead).  

-Motion detection.  It’s a simple enough algorithm to detect movement and only trigger a photo when that occurs.  Yes, this would make it easy to set up a camera to see if the babysitter or the house-painters wander into rooms they shouldn’t, but it would also be very interesting to use with, for example, wildlife photography.

-Allow me to name files sensibly and according to whatever scheme I want.  Right now I rename my images as soon as they’re loaded onto my computer (based on the day/time the photo was taken) but why can’t my camera just do that for me automatically?  And do it according to any scheme I care to specify.

-Let me play with the shutter/flash timing.  Allow me to do multiple-flashes over the duration of a long exposure, for instance, to get interesting multiple-exposure effects.  

-Give me programmatic control over the autofocus and the zoom (if it’s servo-controlled), so I can shoot bracketed focus-points or animate the zoom while the shutter is open for interesting effects.  

-Overlay a reticle on the image, so I can see the framing on different aspect ratios before I shoot the photo.

-Add a nice touchscreen display to the camera and then I’ll be able to easily choose my focus point(s) or choose a region of the image that I want to meter on.

-Activate the microphone to do a voice annotation or capture some ambient sound with every photo.  Or for that matter, allow me to voice-activate the shutter.  (There’s a whole world full of sound-activation possibilities I suspect…)

-Allow me to reprogram the buttons and dials on the camera in any way I see fit.  Any of the buttons/dials, not just a couple of special-case ones…

Some of this hardware control could go down to a very deep level.  Let me play with the scanning rate of the rolling shutter, for instance, to give me super jello-cam images.

And then there’s a whole huge variety of general image-processing operations that could be applied in the camera, from custom sharpening algorithms to specialized color-corrections to just about anything else you currently need to do on your computer instead.  HDR image creation.  Tourist Removal.  Etc.  Are some of these better done as post-processes rather than in-camera?  Sure, probably.  But you could make the same claim about a lot of things – it’s why some people shoot RAW and others are happy with JPEG. Bottom line is that there are times where you just want to get a final photo straight out of the camera. 

(of course the cool thing would be to have the camera create the processed JPEG while also storing the original RAW file.  And then output a metadata file that, when read into Aperture or Lightroom or whatever your RAW processing package of choice is, would duplicate the image processing operations you’d applied in-camera but then allow you to add/remove/modify as you see fit).

And if your camera includes additional useful hardware like GPS or network access or accelerometers then you’ll have even more functionality you can access. Take a look at the iphone app Night Camera which uses the accelerometer to measure when the camera reaches a relatively stable position before taking a photo.  A really smart idea and one that will almost certainly show up in other ‘real’ cameras relatively soon.  (Incidentally, it’s worth noting that the main difficult with algorithmically removing motion blur caused by camera shake is due to the uncertainty about exactly how the camera moved while the photo was being taken – if we were to record accurate accelerometer readings over the duration of the time that the shutter was open, it becomes much easier to remove the blur as a post-process). 

Now, before I get a bunch of replies stating “my camera can already do that” on some of the items listed above, let me reiterate my original point – a camera with the ability to run arbitrary camera-control software would be able to do all of the above items.  And a lot more.  The real fun will start when when lots of people are brainstorming about this and then churn out features that nobody has thought of yet.  Who knows what this might include?

Some of these feature may not feel terribly important or may even seem ‘gimmicky’, but all it takes is a single special-case situation where you need one of them and you’ll be glad they’re part of your software arsenal.  

Back to the iPhone, it’s worth mentioning that we’re already seeing some of these sorts of applications show up.  There are panoramic stitchers and the aforementioned accelerometer function and multiple-exposure resolution enhancers and even a very nice timelapse application.  We may also end up seeing a sort of hybrid solution, with a portable device like the iPhone is tethered to the camera to give the desired capabilities.

I’m convinced that this is where the future of the camera is going, at least from a software perspective* and the features listed above are just a small subset of what we’ll end up seeing.  But you tell me – If you had full software control over your camera, what would you do with it?  

 

 

*The future of the camera from a hardware perspective is a whole other ball of wax but I’ll save my thoughts on that for a later post.  For now I’ll just say that I’m betting that the current paradigm of single (expensive) lens + single (expensive) sensor will eventually be replaced by multiple lenses and sensors and some sophisticated stitching software.  Some sort of Redundant Array of Inexpensive Lenses and Sensors (I call it RAILS).  I’ll try to get a post together on that topic as soon as I get a chance.

Beautiful damage…

I’ve talked before (over at the This Week in Photography blog) about how much fun it can be to rescue old negatives/slides from the obscurity of some box in the basement and get them added into your digital photo archive.  Most of what I’ve scanned so far have come from photos I’ve taken myself but recently (when my Mom moved to a new house) I came across a batch that date from the Mom&Dad era.

…I know that most are primarily of interest only to immediate friends and family but one unique feature of this batch is that several of the original slides were severely degraded by water/mold/dirt/age.  And the patterns that emerged from this slow decay have their own (at least in my opinion) unique beauty as well.  The image at the top of this post is fascinating to me, and I particularly like looking at some of the close-up detail of where the colored emulsions have separated and migrated by the slow action of moisture and mold.

Some of the photos are nothing but damage at this point, others are more of an exercise in trying to decipher the subject matter.

And some feel almost intentional in the way they transition from relative clarity to some impressionistic bit of pseudo-randomness.

(It will be interesting to see if I’ll be apologizing to my mother more for posting such a damaged photo of her, or for posting a photo where she’s wearing those ridiculous glasses…)

Maybe it’s the combination of age, damage and subject matter but I find that many of these images really evoke a strong sense of melancholy.  These next two in particular are almost haunting to me – and in the back of my head I can hear the slow, lonely notes of the minimalist piano score that would accompany them.

I wish I was a skilled enough photographer to be able to capture this sort of thing intentionally.

For those of us who sometimes get caught up in the quest for razor-sharp, high megapixel, ultra-clean digital imagery, it’s all a nice reminder that there are just some things that can’t be quantified.

Bobble

Who’s read Vernor Vinge’s ‘Peace War’ and ‘Marooned in Realtime’? Both books feature this great concept of ‘bobbles’ – spherical stasis-fields that are indestructible, 100% perfectly reflective to the entire electromagnetic spectrum, and where time stops within. Chicago’s Millennium Park features a sculpture called ‘The Bean’ which, when viewed end-on, looks seriously bobble-like. (Particularly if you cheat and stretch the image a little bit on the vertical axis :-)

(Bigger version on flickr)

Subscribe to my Flickr photostream

Just a note, if you want to subscribe to my flickr photostream, you can do it here: http://feeds.feedburner.com/ronbrinkmann_flickr_photostream

The latest images tell the story of the backyard hummingbirds:

Midnight Sun (in Iceland)

(I’d posted this on my homepage a while back but figured I’d bring it into blog-land)

Here’s a time-lapse I took in summer 2006 while wandering around the Northern part of Iceland. Other than the glitch halfway through (apparently a known bug in Canon’s TC80N3 timer, which is otherwise quite cool) it’s a rather nice example of how the sun never sets this time of year. This was shot starting at about 11:30pm and goes through about 2am.

For a much higher quality version of this, check out the quicktime movie(it’s 20MB so it’ll take a while… but it’s much nicer than crappy youtube video.)

And for lots more Iceland photos, go here (and see for yourself what a photographer’s paradise this place is, particularly when you’re able to take photos at about 1am and get stuff like this).

photography, Iceland, timelapse