Vaex & Gaia

The original motivation to develop vaex is the Gaia catalogue containing over a billion stars (at least for data release 1, DR1). This page is meant to get you started with the vaex and the Gaia catalogue.

Getting the data

While it is possible to download all the fits table from the archive and converting them to a single hdf5 file, it is very time consuming and this is already done for you. Download the appropriate file that suits your needs.

Opening the dataset

In [4]:
import vaex
ds = vaex.open("/Users/users/breddels/gaia/gaia-dr1.hdf5")
In [7]:
ds.head_and_tail() # this prints out the first and last 10 rows
#solution_idsource_idrandom_indexref_epochrara_errordecdec_errorparallaxparallax_errorpmrapmra_errorpmdecpmdec_errorra_dec_corrra_parallax_corrra_pmra_corrra_pmdec_corrdec_parallax_corrdec_pmra_corrdec_pmdec_corrparallax_pmra_corrparallax_pmdec_corrpmra_pmdec_corrastrometric_n_obs_alastrometric_n_obs_acastrometric_n_good_obs_alastrometric_n_good_obs_acastrometric_n_bad_obs_alastrometric_n_bad_obs_acastrometric_delta_qastrometric_excess_noiseastrometric_excess_noise_sigastrometric_primary_flagastrometric_relegation_factorastrometric_weight_alastrometric_weight_acastrometric_priors_usedmatched_observationsduplicated_sourcescan_direction_strength_k1scan_direction_strength_k2scan_direction_strength_k3scan_direction_strength_k4scan_direction_mean_k1scan_direction_mean_k2scan_direction_mean_k3scan_direction_mean_k4phot_g_n_obsphot_g_mean_fluxphot_g_mean_flux_errorphot_g_mean_maglbecl_lonecl_latrandom_index_new
0163537841078193356846532610613392014083964353062015.070.4483281598537023.7961224152503883-71.6957489031865696.0268253692814007nannannannannannan0.76905nannannannannannannannannan34034000nan0.00.0701.00.014480601nan214700.88865590.621136250.472959490.69027919-56.079586-61.21716747.14662213.29740168132.746382585192242.834229523121474720.217213325985874283.89010182152231-35.642650089141988314.72479714975469-81.410304906526008453776636
11635378410781933568318300579498427840010642828562015.074.4104350174728351.0220038691630979-10.0980805139362831.705442540935193nannannannannannan-0.7281nannannannannannannannannan75074010nan1.75380193031120761.7725920049008217701.28546260.073237635nan214700.484852640.281430360.210960270.50824386-133.3646433.782207-51.67521344.96596582299.971402372226812.425107979799658419.332070430240847209.11110767897409-29.82493167258337971.705906654981007-32.5526057335091471092218309
216353784107819335684107765137787303424126659832015.0258.985589298849566.4453302404811881-27.8824175471040723.6298806298255251nannannannannannan0.19715001nannannannannannannannannan62059030nan3.53479227382015716.7357655148722697013.6442480.028884925nan27700.684414680.481574180.916598260.81429851-59.671658-26.367931-31.970381-40.6387957239.19849856679965.432544030139473219.577873940622769357.30089831414876.0352140807129935260.24276152199383-4.8154409556136368228972814
31635378410781933568600090164282164326411106052372015.0229.907910730740640.76535752248689992-42.8755600165940860.37139504470037371nannannannannannan0.61825001nannannannannannannannannan1230122010nan0.00.0701.00.32922804nan217700.367619840.564124460.287940560.51157725-8.0175171-10.897376-30.979944-34.078159138496.132561937798982.059084351980613518.785775735386174329.7798218451671312.139534343431455238.98533766350926-23.65689127625858137329428
41635378410781933568599183967414435353610792115032015.0241.598656952218310.53098435671902211-43.4221180297244590.29834024425635697nannannannannannan0.76709998nannannannannannannannannan3150315000nan0.706079910060678384.8325400359917579701.19040350.53842324nan244700.610652690.83471560.573947430.52250642-0.81963819-3.96563550.86793053-2.0025661338998.026782900814511.85975419353585318.026914573573116336.263891361941946.4807110333809845248.10500341825389-22.1178229430088861104491361
5163537841078193356821979777574379677444555201322015.0331.309267191268020.9919430885678801655.764175124104981.1849859537856224nannannannannannan0.82620001nannannannannannannannannan1580158000nan2.45666557308826946.6083453891993509702.60111740.062909283nan224700.559321110.691776220.767197070.85186577140.256-43.12843713.977358-41.003326173321.165867208489713.182217235249227319.257946605813117101.021260594612580.129375924323943519.325921425540332959.990928012054368921427467
6163537841078193356853645028008301564167641022772015.0160.478247259642810.5922876483566184-48.5190544538378530.45834025661358119nannannannannannan-0.84375nannannannannannannannannan1660166000nan0.923216673481246926.4692720469718452701.22169650.27006039nan226700.428105650.478981760.59552240.8984310617.29542930.46129244.12918133.785625181649.436644940913652.685214058516274718.493428087922247281.884370554096398.9878214290922749188.646030882117-50.841304973621433530870392
716353784107819335685144355833950611208584810232015.037.8037605165771571.219763409374453863.9930141686374012.0458496306270808nannannannannannan-0.90654999nannannannannannannannannan1360134020nan0.00.37561602019322371701.00.045570314nan225700.388118090.651192490.19253540.6802154880.07333459.38356451.201408-32.894348143157.677524008659961.796110987721647220.030345584714247133.60551488977683.208577615375917860.16544230219773945.862248114935028817271282
8163537841078193356840506297335215467521169965652015.0273.934662658152389.3794694752461023-28.5022161716737839.7884171083320464nannannannannannan0.99914998nannannannannannannannannan54054000nan1.02161223293971955.5164919128030521701.59531590.50649303nan26700.446690080.342838230.537837980.98528355-92.730103-40.863827-32.72839442.657158541851.00983197664486.574066230536940817.3562482498568973.6482572846267969-5.5235762470358933273.47098392251576-5.1103309217573605526416604
9163537841078193356867360404815662112005217596312015.0282.2885142402300319.891129161664466-32.71912314885305123.421473514841974nannannannannannan0.99954998nannannannannannannannannan51051000nan0.00.0701.00.20831257nan210700.647099550.998177770.64741910.99271953-46.968323-48.301346-46.97082141.69908150307.948068703821892.261686614464766319.3035763519476372.9747377414579947-13.82795742055284280.46735228599545-9.7252828107948233136200710
..............................................................................................................................................................................
1142679759163537841078193356859638181092181149441642522142015.0256.1640587442608413.195403515463424-43.97069741858192112.094350109113678nannannannannannan0.87599999nannannannannannannannannan43042010nan3.80486263419194512.30467268162293287014.6813140.023070171nan27700.740398470.241699140.684454980.99584007-98.52498654.358448-20.61586-40.37592750136.010298105666292.863170880263776320.190840582599829342.96637630841758-1.6522439126765152259.37359480316428-21.040478760992286363507518
11426797601635378410781933568607465025568930803210765418942015.0189.0942853219818931.128472777182477-53.70620974670751515.333894829015923nannannannannannan0.90724999nannannannannannannannannan60060000nan6.22348313377098352.05650942583363697023.9788110.0066079721nan212700.831036810.906047280.967283790.79334909-64.38958-56.39530658.99417930.51685976131.374325204787342.816529386534308220.228493818045031300.675839480461959.0995533348250301214.81433551529065-44.609231471802872522780454
11426797611635378410781933568411496903055928179210864800822015.0259.335530127241211.6556817140665616-21.9644114441567310.97959959433487354nannannannannannan0.98140001nannannannannannannannannan67067000nan0.6760582855296107814.15949494779532701.99062721.5665698nan28700.434982240.372675870.658469740.82623446-86.497498-22.893591-28.382563-41.679073667079.507158334502516.03925021142314815.8997625007834082.39286918983255839.1423448773656553260.115821689822551.1096985416795939443656666
1142679762163537841078193356833864290525792933124325113002015.0100.584942091374676.835607564781991428.1176526446082817.3412944028577414nannannannannannan-0.99870002nannannannannannannannannan52051010nan0.629565429720549411.427477534995772701.22879830.70854688nan27700.428230490.302920670.543806370.98244995-81.41381141.421326-28.360426-41.469437511504.80300288961267.303007031824676317.581070940856129186.6403358485828810.58795256110684999.3600700476713855.02165122018746591070554463
1142679763163537841078193356822117507254154707206062110622015.0342.725809939805860.7402987746820737265.8173302982038850.62560998734695905nannannannannannan0.29949999nannannannannannannannannan72072000nan0.00.0701.00.23586625nan216700.59235930.150165070.229407190.401998437.409813-12.897958-23.375769-22.76032198506.895725117932954.10279394797959118.762473492250376111.002858192314755.777072168183858432.71434391113349762.295141067332615503242256
11426797641635378410781933568585436959074758118410165002982015.0216.805303172198680.91630109508397251-61.7246998700547490.39551128668858176nannannannannannan0.64660001nannannannannannannannannan1070107000nan1.25328356052779344.4618355665047806701.46912530.22151491nan219700.270455930.333502470.118910750.34374031-34.48344-14.086372-44.686737-42.768448115679.910814842901692.546331564034801418.443640190758032314.01842346058885-0.93214994153602959238.1566176380735-44.03689421835278589408763
1142679765163537841078193356858773371904287485445715243912015.0220.183933962477511.4540088365312194-62.0693809841198670.66605780363196576nannannannannannan0.16345nannannannannannannannannan1240123010nan0.00.0701.00.070951678nan219700.383593230.145812510.0850696710.16669448-66.7919920.5530284-28.570841-43.504219131193.679148384005631.853369687198007719.807062896777158315.354706269823-1.8630425403260276240.35340816799865-43.660839433284728190647637
1142679766163537841078193356840981690469353877763401329412015.0275.592829744740197.5966921147437994-15.1864234936372217.5496371354566678nannannannannannan0.99615002nannannannannannannannannan89088010nan1.37265246474536241.4436194912075431705.37795930.085397005nan211700.400080260.627909660.540180860.94192868-69.717201-40.794945-36.37086543.89116386251.624831729589911.964893740907249619.52288632016865616.144559733458344-0.61930949509451483275.452134280846688.1470700642669698441793051
114267976716353784107819335684096373888078681600533117952015.0275.253433424473434.0775107054738715-17.9984294368576384.0513048860510494nannannannannannan0.99715nannannannannannannannannan63063000nan0.9471344778573754813.458668006207617703.78074030.7676605nan28700.563734890.719114720.653652610.96535587-56.763374-42.042133-38.707329-44.976742592951.96811019030886.36350360248786216.84949090999689113.509039465141568-1.6537669044298298275.017609712258095.34939581612028154920551
11426797681635378410781933568403747318770866048010367240572015.0269.9059981255110210.126195176264476-35.8818357479608359.567640863968446nannannannannannan0.99954998nannannannannannannannannan53052010nan0.580418345547981223.3654171905030141702.44794991.2009112nan27700.651207450.988912820.630209620.95600069-41.550079-42.252548-41.553543-42.278587532248.84977622260484.790943807850289317.144868950303255355.49758724701445-6.0628355228565223269.92201797155531-12.4425803563719021219805

First plots

In [10]:
# make sure plots go inline
%matplotlib inline
In [11]:
ds.plot("ra", "dec", f="log10") # sky plot, showing log10 of counts
Out[11]:
<matplotlib.image.AxesImage at 0x7f74e02772e8>
_images/gaia_5_1.png
In [21]:
# The gaia data is recognized, no healpix expression needed
ds.healpix_plot(f="sqrt", figsize=(10,7))
_images/gaia_6_0.png