# Variability and new-source detection stage¶

## Variability index calculation¶

After all images for a given timestep have been processed and the resulting source measurements have been assigned to runningcatalog entries (effectively lightcurves), variability indices are calculated for the most recent timestep, and stored as part of the association recorded in the assocxtrsource table.

Note that a single runningcatalog source may contain entries from multiple independent frequency bands. The variability indices are calculated independently for each frequency band, hence the $$\nu$$ suffix in the calculations below denotes an index over the different bands.

For a comprehensive discussion of the transient and variability detection algorithms currently being employed, see Scheers (2011) chapter 3. Here, we provide a brief outline.

We define two metrics for identifying variability in a lightcurve. The flux coefficient of variation, which we denote $$V_\nu$$, is defined as

$V_{\nu} \equiv \frac{s_{\nu}}{\overline{I_{\nu}}} = \frac{1}{\overline{I_{\nu}}} \sqrt{\frac{N}{N-1}(\overline{I^{2}_\nu} - \overline{I_{\nu}}^2)}$

where $$\overline{I_{\nu}}$$ is the mean flux of all measurements in the lightcurve at frequency $$\nu$$, $$s_{\nu}$$ is the standard deviation of those flux measurements and $$N$$ is the number of measurements.

The second metric is $$\eta_{\nu}$$, which is defined based on reduced $$\chi^2$$ statistics as

$\eta_{\nu} \equiv \chi^{2}_{N-1} = \frac{1}{N-1} \sum_{i=1}^{N} \frac{(I_{\nu,i} - \overline{I_{\nu}}^*)^2}{\sigma_{I_{\nu,i}}^2}$

Where $$\overline{I_{\nu}}^*$$ is the average of the flux measurements weighed by their uncertainties. $$\eta_{\nu}$$ is the $$\chi^{2}$$ probability distribution. The probability that the source is “flat” (i.e. has no significant variability) is then the integral of the distribution from the measured value of $$\eta_{\nu}$$ to $$\infty$$; the probability that it isn’t flat is thus 1 minus this quantity.

See also the appendices on the database schema for details of how these are iteratively updated.

## ‘New source’ detection¶

We also attempt to identify any newly extracted sources which we suspect are intrinsically variable in nature (i.e. they are getting brighter, as opposed to our observations getting deeper or even simply looking at a previously unobserved patch of sky). The algorithm for evaluating new sources is encoded by tkp.db.associations._determine_newsource_previous_limits(). Sources we deem to be intrinsically ‘new’ are then recorded in the newsource table.