Cracked it: automatically adding files to iTunes without copying

Yep, finally did it! Has been bugging me for years!

Click here to skip directly to the solution for adding files to iTunes without copying

So, this is the story: I have iTunes and in spite of it’s faults, it’s excellent at organizing music and setting metadata. It’s also the only reasonable way to sync up your iOS devices or Apple TV with new stuff. Problem is, once we get into video, one iTunes setting becomes an issue: “Copy files to iTunes Media folder when adding to library“.
Screenshot: copying files to iTunes

Many people keep this setting on, because it means iTunes will automatically add songs to the Music folder and put them in the right album folder, etc. Bascially, the files will be copied to your iTunes directory. But when you start adding video files in gigabyte sizes, often stored on external drives, you don’t want to fill up your drive with copies, you’d prefer to have them on the external drive.

There is one simple way to add files “by reference” (e.g. not copying but referring to original location) by simply following this MacWorld hint: basically, hold the Option key while dragging a file to the iTunes window (not the dock icon, and the Command key is not needed in spite of the hint suggesting so).

Problem is, that is easy to forget to do. When you have lots and lots of videos, you may also want to use scripts to automatically add the right kind of files to iTunes. So I went out searching for how to programmatically – in a script – add files as references to iTunes. Turns out it’s not so easy.

First approach was to look at the AppleScript API for iTunes. Following the famous Doug’s AppleScripts for iTunes guide seemed to be the right place to start. There I learnt that I can add files to iTunes by running the AppleScript:

tell application "iTunes"
set newFile to (choose file with prompt "Select a song to add...")
add newFile
end tell

You can replace the “choose file” prompt with a programmatically created file path. Unfortunately, this adds the song according to the “Copy files to iTunes…” preference, but not as a reference.

Second attempt: I couldn’t find a way to emulate the drag ‘n drop in the MacWorld tip using AppleScript, but what I found out is that you can also go to File > Add to Library while holding Option key, and pick your file while still holding the Option key. That will add the file as a reference, not as a copy.

So how to automate? AppleScript has commands like:

tell application "System Events"
key down option
key up option
end tell

… but it all failed on the case that “Add to Library” command would open a choose file box which makes automation impossible.

So what did I finally find? It’s a bit weird. As described in How to move files into iTunes instead of copying, there is a folder called “Automatically add to iTunes” inside the iTunes music folder. Anything you drop there will be added immediately to iTunes. But now I wanted to add a reference to a file elsewhere, not just move the file itself into iTunes. It turns out that if I drop an OS X alias file (e.g. a shortcut) pointing to the original file, iTunes will add it as a reference!

Two peculiarities to note:

  1. Create an alias file by right-clicking and choosing “Make alias” or by drag ‘n drop while holding Shift+Option+Command. Unix symlinks, that otherwise work in OS X, will not add the file as a reference, so avoid them for this purpose.
  2. Once the alias file has been added, iTunes will fail to remove it (normally, files dropped in the “Automatically add” folder will be moved into the iTunes structure). That means that iTunes will see this file over and over again – on your next drop in the folder, iTunes will add both the new file, and the old leftover file again. So you need to manually remove the alias as soon as it’s in iTunes.

But what about scripting? Well, try these two lines on the OS X terminal:

osascript -e 'tell application "Finder" to make new alias file to POSIX file "/path/to/original.file" at POSIX file "/Users//Music/iTunes/iTunes Music/Automatically Add to iTunes"'
rm "/Users/
/Music/iTunes/iTunes Music/Automatically Add to iTunes/original.file"

Replace the paths to the ones correct in your system, of course. The two lines can easily be added to any shell script or python script, which could then process for example your most recently downloaded video files.

I will be adding this feature to a soon finished script that automatically organizes your video collection, so stay tuned!

iTunes silently not importing files

After using the simplest ever method for adding files to iTunes, I kind of forgot the whole importing thing – it’s automatic. But I recently reminded myself of the need to regularly check the “Automatically Add to iTunes” folder for any files that were rejected.

What I saw was that more than 15 MP3-files had not been imported. I tested them and they played fine on VLC, even in QuickTime Player. But in spite of trying all possibilities of adding them to iTunes, they were ignored. Weird.

Something in the files made iTunes throw up. I never figured out what, but I figured out a solution. I went to Mac App Store and downloaded Music Converter (free version). When I drag and drop an audio file onto it, it will do any necessary conversion and send it to iTunes. It can even make Apple Lossless, FLAC, etc! In this case, the conversion was instantaneous, hinting at there was something in the MP3 container or ID3 tags that were wrong (rather than a more time consuming codec conversion / transcoding), but that Music Converter could fix. Now all my songs are imported in iTunes!

Inofficial ultimate FAQ for iTunes Match

iTunes Match launched recently, the $25 service where Apple will let you keep your music in the cloud for access on any device. They have chosen a slightly different approach than the typical musical lockers, that I find quite interesting. (read more from Macworld and Walt Mossberg) But many also find it confusing, so I have tried to collect all frequently asked questions around the web and collate them here. Feel free to share new answers, corrections or questions with me using the comments field!

Q: Is Match a streaming service?
Yes and No. If a song is not on your device but in your iCloud library, if you click it, it will start downloading AND stream it (the file will most likely download faster than you can listen to it). After that, the song will be stored on your device in case you want to play it again offline. There are conflicting reports whether the music is cached (will be deleted after some time) or stored (risking filling your device after a long time) (MacRumors) (Walt Mossberg).

Note that Apple TVs and iTunes on computer are different from iOS devices – they will not download, only stream music (possibly using a hidden cache). (MacRumors)

Q: What is the exact difference between Match and a streaming service like Spotify or Google Music?
a) iTunes Match streams but downloads the files in the same time and keeps it for a longer period of time, whereas most streaming services would either ONLY stream, or stream and cache for a short time. So with the exception of managing the space on your device, in this regard Match is an improvement to what Spotify and Google Music does.

b) Most streaming services give access to their complete library (e.g. Spotify), but Match only gives you access to YOUR complete library.

Q: What will happen to my current songs on my iOS device?
Nothing, really. There will be prompt when activating on your iOS device that may seem like your old tracks will be deleted, but they are not – by definition if the tracks are on your iPhone, they are also in iTunes, and as such will also be in iCloud. After activating Match on the device, playing a song that you had before should play the copy that you had previously synced. Playing a new song would download it from iCloud, normally as 256kbps. Tip: If you try to upgrade the quality of your tracks to the Match 256kbps AAC standard, follow this procedure to get them on the iOS device, as it may otherwise be playing your original, lower-quality copy. Also check this article out.

Q: What happens to music on my non iOS iPods such as iPod Classic, Nano or similar?
Nothing. They don’t have network access and therefore cannot use iCloud, so they will be synced as normal with iTunes.

Q: Will my metadata be cleaned up when matching?
No. To avoid destroying any custom metadata used, Match will copy all YOUR metadata to the matched iTunes Store song. So if your metadata is different from iTunes Store metadata, YOURs will be used. That unfortunately means that you CANNOT use Match to clean your Metadata. (under investigation)

Q: What is the maximum amount of songs that I can have in Match?
If the number of songs in your library, excluding ones you have previously bough from iTunes, are more than 25000, Match will refuse to be used at all. There are workarounds, however. (Note that, 25000 songs is something like 125+ GB of music depending on your music quality and song length, so most people are probably well below. You can see the number by looking in the bottom bar of iTunes after selecting “Music” from the sidebar).

Q: What happens with songs that iTunes Match cannot find/match in the iTunes Store?
They will be uploaded exactly as they are in your library. This is the same behavior as Google Music or Amazon Cloud Drive. (Match just adds the ability to recognize songs already in iTunes Store to avoid uploading them unnecessarily).

Q: What exactly will be matched and/or uploaded by iTunes Match?
It will match any iTunes content marked as “Music” that has a bitrate higher than 96kbps and are smaller than 200 MB.
Tip: You can fool Match to upload low bitrate songs by selecting “Convert to AAC” on them. The converted version will have a (fake) higher bitrate (but will also take a little more space), but will be matched or uploaded. Once it’s matched, you should be able to delete your local (size inflated) copy. (under investigation, there are some conflicting reports whether it matches higher than 96kbps or equal and higher)

Q: What is the quality of matched songs?
It is 256kbps AAC iTunes Plus songs, completely DRM free, same as if they were bought from iTunes Store. That should also presumably mean that they have been created from digital masters (24bit/96Khz) which should result in a better quality and no errors compared to your own rips (note that it’s possible the 256kbps AAC version could sound better than e.g. 256kbps MP3 or higher, but you better test listen first – most likely you won’t see a difference).

Q: Can I use iTunes Match to upgrade the quality of my music library?
Yes! All songs that are matched can be downloaded by you first removing the original, then re-downloading it from iCloud. The new version will retain the old metadata but be in 256kbps AAC. This effectively enables you to remove all your low-quality files from the library. If you want to upgrade your songs in batch, go ahead and read this Macworld article.

Q: Will iTunes Match use my 3G download caps or load the network a lot?
iTunes Match will download and stream songs on 3G as well as on WiFi. Assuming you download your major playlists when you’re on a WiFi (typically after you have run iTunes Match), the access over 3G should be quite limited and would over time use less bandwidth than a streaming service (because the streaming service wouldn’t save songs for a longer time).

Q: Will Match use up more battery?
Yes, it will use a bit more battery than before, but only when you download you new songs. So it depends on your behavior – if you download a big playlist first and only listen to that, there is no extra load on the battery.

Q: Can I listen to music when having no network such as in airplane mode?
Yes of course, but you will not be able to download any new songs, so what you can listen to depends on what you have previously downloaded to the device. Before going offline for a longer time, make sure you download your favorite playlists from the iOS device, preferably when you are in your home WiFi.

Q: Will iTunes Match risk filling up my device with music so it becomes too full to use?
(under investigation)

Q: How is a song matched by iTunes Match?
The exact method is not yet identified, but it’s all pointing to a combination of audio fingerprinting and possibly metadata. Fingerprinting means something similar to SoundHound and Shazam, which doesn’t work in all cases (depending on music, quality of rip, etc). Metadata may enhance this but it would not be enough to use metadata alone – if it did anyone could name their songs to anything they want and match it.

Q: Why are so many songs not matched by Match?
First, songs not available in iTunes Store will not match, which will typically be more obscure music, indie, live recordings, etc. In the case that you can find the song in iTunes Store but it is still not matched, it can be two reasons:
1) your metadata and/or song is too different from the iTunes version
2) Apple has not secured a deal with the copyright holder of that song.
However, non-matched songs will still be uploaded to the cloud, so the only effort needed is the initial upload

Q: What happens if I end a subscription of Match?
You will not be able to access your iCloud anymore, but any songs you have downloaded from Match will stay where they are. So if you have replaced your lower quality songs with new from Match, they will still be retained. (under investigation)

Q: Can I shuffle among songs not yet downloaded and songs downloaded? E.g. can I use iCloud as a giant jukebox?
(under investigation, probably yes)

Q: What happens to my lossless songs?
If they are matched, they will be represented in iCloud by a 256kbps AAC song, and subsequently used on your other devices. If they are not matched, a 256kbps AAC song will be temporarily created from the lossless song and uploaded. Only your main computer, where the lossless songs originally were matched, will have the lossless songs. This means may mean that if you were previously listening to Apple Lossless on your iPhone, it would now be in 256kbps AAC. It remains to be discussed whether this difference is noticeable on an iPhone.

Q: What happens to my (intentionally kept) mono, original master, explicit songs or bonus tracks?
There is a risk that iTunes Match will pick a song that is stereo, remastered or non-explicit. It has been shown that Mono matches to Stereo songs. Others say that Explicit versions are kept intact. Bonus tracks on CDs are often padded onto the last track after a period of silence – in iTunes the bonus part is treated as a different track, so you may not be able to match those tracks properly from a CD rip. However, nothing on your computer will be replaced, it’s just that your other devices will get the iCloud version rather than your original. (under investigation)

Q: Will iTunes Match track my illegal / pirated songs or report me to RIAA and others?
Obviously, no one knows what secret stuff Apple could be doing, but it is highly unlikely. They would know that a large amount of Match users’ songs have dubious origins, and that fact must have been discussed when they agreed on the deals with the music labels. Also, it’s very difficult to prove that a song is downloaded or ripped from your own CD just by looking at the file, what they normally need to prove is the downloading itself with IP-address. Note however that matched songs will get metadata that links it to your account – if you were to fileshare downloaded matched songs they could be traced to you unless you cleaned the metadata first (and that still does not exclude hidden fingerprinting of the songs).

Q: Can I hide or choose not to have some songs in iCloud?
Not normally – although your device may not have downloaded the song it will display it in the library. To make sure it’s not there, you have to permanently delete it from Match but then it would also not be in your main library.

Q: What happens to my purchased songs?
If they were purchased the last few years when iTunes Store became DRM-free, they will already be in your iCloud as previously purchased music so no need to do anything with iTunes Match. If you have old purchased songs with DRM enabled (protected AAC) they will be kept as is. If you explicitly want to “un-DRM” them, it’s possible – see this article from Macworld. When it comes to songs that were purchased from other Apple IDs, they may be recognized as any ripped song (e.g. not counted as a purchased one). (under investigation)

Q: If a song was not matched and uploaded instead, can I later try to re-match it, e.g. if something changed in matching algorithm or song availability?
(under investigation)

Q: Will a matched song retain playcount, date added and those iTunes specific meta data?
(under investigation)

Q: What does the iCloud icons in iTunes and on iOS mean and how can I see the status?
See here. You can also in iTunes select View > View Options > iCloud Status & Bit Rate. They will show in a column in iTunes with a little more information.

Q: I am not in US, what does this mean for iTunes Match, can I use it?
Currently, it is only offered to Apple IDs in the US Store that has a US credit card attached to them (accounts made withouts credit card will not work). Matching will presumably be done to US iTunes Store only, meaning that songs not available there will not be matched (but instead uploaded). If you have a US credit card on your Apple ID, you can sign up and should not have any problems accessing the service from another country (see below).

Q: Assuming I have successfully signed up for Match, can I use it abroad?
Yes, you can. Apple only checks the iTunes account and credit card country, not the country you are accessing from. You may have slightly worse performance abroad (e.g. more buffering, slower download) – it depends on to what extent Apple uses CDN for Match.

Q: Can I pay for the subscription of Match with a gift card?
Yes, if you have gift card balance you can use that, but you still need a US credit card registered to the account.

Q: What happens to playlists I had before or create later on my computer?
Any computer or device with iTunes Match enabled will instantly upload and sync all playlists with iCloud. That means the playlists will be viewable on all devices, but it does not mean all their songs are downloaded automatically – songs will be downloaded when the user plays them or manually chooses to download them.

Q: What happens if you share your iTunes store account with multiple family members?
(under investigation)

Q: How does iTunes Match handle multiple iTunes libraries?
(under investigation)

Q: How does Match treat album art, e.g. embedded vs. iTunes downloaded?
(under investigation)

Q: Will my 5GB of iCloud storage be affected?
No iTunes Match songs will NOT affect your iCloud storage limit, e.g. both purchased, matched and uploaded tracks will be outside of this quota.

Q: How can I completely delete a song?
When deleting from iTunes, also check the box to delete it from iCloud. It will be thrashed as normal on your computer, but it will also be taken off iCloud and deleted on all your connected devices.

Q: I currently downsample my songs to 128kbps when syncing to iPhone / iPod Touch, what will happen after activating Match?
All songs on iCloud will be sampled at 256kbps if matched or if a higher bitrate song was marked for upload, or else it will be sampled at the original bitrate if it was uploaded. That means your iOS devices will access and download the 256kbps versions in many cases. So this will sound a bit better but also take a bit more space on your device. On the other hand, you can delete songs from your device and still have them re-accessible in the cloud. Note that unless you delete the 128kbps songs from the device after activating iTunes Match, the 128kbps versions should still be used. It will only download the new 256kbps copy when there is no pre-existing song on the iPhone / iPod.

Q: Will iTunes Match work on PowerPC, Leopard, Snow Leopard and similar?
Yes, in spite of the Apple article stating otherwise users report that they are running iTunes 10.5.1 and Match on Leopard or Snow Leopard successfully. However, the rest of the iCloud syncing (such as documents) seem to require OS X Lion.

Q: Can I download podcasts from the internet from my iOS device?
(under investigation)

Q: Does anyone know what iTunes Match does when you want to rip a new CD? Does it just skip the whole rip process, and analyze the CD directly for matches, or does it make you go through the process of ripping each track and then match the files afterwards?
(under investigation)