ekos scheduler

Introduction

Ekos Scheduler is an indispensable arsenal in building your robotic observatory. A Robotic observatory is an observatory composed of several subsystems that are orchestrated together to achieve a set of scientific objectives without direct human intervention. 

It’s recommended to use Ekos Scheduler after you are familiar with using all the Ekos modules manually first. Fine-tune the settings for each module to suit your particular equipment setup.

Settings

Ekos Scheduler provides a simple interface to aid the user in setting the conditions and constraints required for an observation job. Each observation job is composed of the following:

  • Target name and coordinates: Select target from the Find Dialog or Add it from Observation Planner. You can also enter a custom name.
  • Optional FITS file: If a FITS file is specified, the astrometry solver shall solve the file and use the central RA/DEC as the target coordinates.
  • Sequence File: The sequence file is constructed in the Ekos Capture Module. It contains the number of images to capture, filters, temperature settings, prefixes, download directory..etc.
  • Priority: Set job priority in the range of 1 to 20 where 1 designates the highest priority and 20 the lowest priority. Priority is applied in calculating the weight used to select the next target to image. (Only enabled in Classic algorithm)
  • Profile: Select which equipment profile to utilize when starting Ekos. If Ekos & INDI are already started and online, this selection is ignored.
  • Steps: Each job goes through a sequence of discrete steps. Each step or stage can be toggled on or off as desired:
      1. Track: Mount is commanded to slew to target.
      2. Focus: Camera autofocus (if applicable) is started.
      3. Align: Plate-solving is performed to ensure the correct location, framing and orientation of the target is met. If a FITS file is specified in General Settings, then this file is first plate-solved and then mount is commanded to to slew to target solution coordinates. This is followed by another plate-solving process to ensure we are within tolerance at the target solution coordinates. If the position angle of the FITS image is different from the current camera orientation, the camera orientation can be automatically adjusted if a mechanized rotator is detected. Otherwise, a manual camera rotation is required until the image position angle is satisfied.
      4. Guide: Using a guide camera, the mount tracking is locked to a guide star to enable long-exposure astrophotography
  • Startup Conditions: Conditions that must be met before the observation job is started. Currently, the user may select to start as soon as possible Now, or when the target is near or past culmination, or at a specific time.
  • Constraints: Constraints are conditions that must be met at all times during the observation job execution process. These include minimum target altitude, minimum moon separation, twilight observeration, and weather monitoring.
  • Completion Conditions: Conditions that trigger completion of the observation job. The default selection is to simply mark the observation job as complete once the sequence process is complete. Additional conditions enable the user to repeat the sequence process indefinitely or up until a specific time.

You must select the Target and Sequence before you can add a job to the Scheduler. When the scheduler starts, it evaluates all jobs in accord to the conditions and constraints specified and attempts to select the best job to execute. Selection of the job depends on a simple heuristic algorithm that scores each job given the conditions and constraints, each of which is weighted accordingly. If two targets have identical conditions and constraints, usually the higher priority target followed by higher altitude target is selected for execution. If no candidates are available at the current time, the scheduler goes into sleep mode and wakes up when the next job is ready for execution.

scheduler planner

The description above only tackles the Data Acquisition stage of the observatory workflow. The overall procedure typically utilized in an observatory can be summarized in three primary stages:

  1. Startup
  2. Data Acquisition (including preprocessing and storage)
  3. Shutdown

Startup Procedure

Startup procedure is unique to each observatory but may include:

  • Turning on power to equipment
  • Running safety/sanity checks
  • Checking weather conditions
  • Turning off light
  • Fan/Light control
  • Unparkig dome
  • Unparking mount
  • ..etc

Ekos Scheduler only initiates the startup procedure once the startup time for the first observation job is close (default lead time is 5 minutes before startup time). Once the startup procedure is completed successfully, the scheduler picks the observation job target and starts the sequence process. If a startup script is specified, it shall be executed first.

Data Aquisition

Depending the on the user selection, the typical workflow proceeds as following:

  • Slew mount to target. If a FITS file was specified, it first solves the files and slew to the file coordinates.
  • Auto-focus target. The autofocus process automatically selects the best star in the frame and runs the autofocus algorithm against it.
  • Perform plate solving, sync mount, and slew to target coordinates.
  • Perform post-alignment focusing since the frame might have moved during the plate solving process.
  • Perform calibration and start auto-guiding: The calibration process automatically selects the best guide star, performs calibration, and starts the autoguide process.
  • Load the sequence file in the Capture module and start the imaging process.

Shutdown

Once the observation job is completed successfully, the scheduler selects the next target. If the next target scheduled time is not due yet, the mount is parked until the target is ready. Furthermore, if the next scheduled target is not due for a user-configurable time limit, the scheduler performs a preemptive shutdown to preserve resources and performs the startup procedure again when the target is due.

If an unrecoverable error occurs, the observatory initiates shutdown procedure. If there is a shutdown script, it will be executed last.

The following video demonstrates an earilar version of the scheduler, but the basic principles still apply today:

 

The Greedy Scheduler

Greedy Scheduler will allow you to choose a new scheme for scheduling jobs. (In the nightly release) there should now be a scheduling algorithm choice. If you set it to "Classic" (the default), nothing has changed. If you set it to "Greedy" you will see the changes described below.  Note: these changes concern scheduling--deciding which jobs run and when. They do not affect the job management aspects of the scheduler which remain unchanged.

In both the Classic and Greedy schedulers, jobs are listed as "earlier on the job list means higher priority". With the Classic scheduler, priority** is of the highest importance. It will not schedule a lower priority job until the higher priority job is done, even if that job takes several nights, and even if the higher-priority job cannot run at the current time, e.g. due to altitude/terrain/etc.  In contrast to this, the Greedy scheduler attempts to keep Ekos busy as much as possible. Although it gives priority to earlier-listed-jobs, it will run later-listed ones if the earlier one can't run. Of course, the lower priority job will get preempted when the high-priority job can finally start to run.

**In this discussion, "lower priority" means "further down on the job list". and similarly, higher priority is higher/earlier on the job list. We will be phasing out the priority number that can be assigned to jobs on the Classic scheduler.

If the scheduler is started with only one job, there is no difference between Classic and Greedy scheduling. However, if you have more than one job, depending on your setup, there is a good chance that the Greedy scheduler will schedule more imaging time than Classic.

Here's a recommended way to try this out. Let's assume you have a main target for which you want to collect as many images as possible. Set that target up as first on the scheduler list, have it start ASAP and set its completion condition as "Repeat Until Terminated". It should be scheduled to image whenever possible (even across multiple nights) until you turn off the scheduler or Ekos. Add several other targets as well, ones that you might also be interested in, and that can be imaged in other parts of the night. Make sure those are listed below the primary target on the jobs list. Set those the same way (ASAP/RepeatUntilTerminated). They will be scheduled to run whenever the primary target can't be imaged.  Of course, make sure the twilight restriction is set for all your targets. Altitude and terrain restrictions are important as well--if Ekos doesn't know that there's a tree or house blocking your target, it can't be smart about scheduling it.

Since jobs will be preempted/restarted more often with Greedy than with Classic, the "Remember Job Progress" option is now more important. Can find this setting in the KStars Setting Menu --> Ekos --> Scheduler --> "Remember job progress".  This option only works if you are storing images on the same machine where the scheduler is running. Should enable "Remember Job Progress" with this scheduler to get the most benefit, assuming your capture sequences use multiple/different filters. If your capture sequences are just used with one type of filter/or OSC then it probably doesn't matter. (RememberJobProgress has also be updated a bit, and should now do a better job of picking up where the last attempt finished.)

Here is a screenshot of the scheduler running with this new scheme. Note that there are 4 jobs, and they are listed in order of precedence. See the next start times for each of the jobs on the scheduler table, and the highlighted (4th) job is the one currently running. A schedule for the next 48 hours is also printed in the log window at the bottom.

greedyScheduler2

The scheduler is (minimally) integrated with Analyze. Here's a screenshot of the Analyze timeline for the 3 days. (Note some timelines didn't display well because of screen resolution.)  The top line is the new scheduler timeline in Analyze. The different colors correspond to different jobs that were run, each one keeping its same color on different stars (yes, red is an unfortunate choice of colors and I've already submitted code that changes the color scheme for that line). Different jobs can be seen ran as expected each night.

Greedy3dayB

Weather Monitoring

Another critical feature of any remotely operated robotic observatory is weather monitoring. For weather updates, Ekos relies on the selected INDI weather driver to continuously monitor the weather conditions. For simplicity sake, the weather conditions can be summed in three states:

  1. Ok: Weather conditions are clear and optimal for imaging.
  2. Warning: Weather conditions are not clear, seeing is subpar, or partially obstructed and not suitable for imaging. Any further imaging process is suspended until weather improves. Warning weather status does not pose any danger to the observatory equipment so the observatory is kept operational. The exact behavior to take under Warninig status can be configured.
  3. Alert: Weather conditions are detrimental to the observatory safety and shutdown must be initiated as soon as possible.

Aborted Job Management

Define what should happen when a job steps into an error or aborts:

  • Don't re-schedule (None): Don't restart the job in case of an error or an abort.
  • Re-schedule after all terminated (Queue): If a job gets aborted, the scheduler will only re-schedule it if when all jobs are finished or aborted. If this is the case, the scheduler re-schedules all aborted jobs and sleeps for the given delay.
  • Re-schedule immediately (Immediate): As soon as a job gets aborted, the scheduler will re-schedule it and waits the given delay.

If the option for re-scheduling errors is selected, errors are handled like aborts. Otherwise, jobs that step into an error are never re-scheduled.

Startup & Shutdown Scripts

Due to the uniqueness of each observatory, Ekos enables the user to select startup and shutdown scripts. The scripts takes care of any necessary procedures that must take place on startup and shutdown stages. On startup, Ekos executes the startup scripts and only proceeds to the remainder of the startup procedure (unpark dome/unpark mount) if the script completes successfully. Conversely, the shutdown procedure begins with parking the mount & dome before executing the shutdown script as the final procedure.

Startup and shutdown scripts can be written any language that can be executed on the local machine. It must return 0 to report success, any other exist value is considered an error indicator. The script's standard output is also directed to Ekos logger window. The following is as sample demo startup script in Python:

#!/usr/bin/env python
# -*- coding: utf-8 -*-

import os
import time
import sys

print "Turning on observatory equipment..."
sys.stdout.flush()

time.sleep(5)

print "Checking safety switches..."
sys.stdout.flush()

time.sleep(5)

print "All systems are GO"
sys.stdout.flush()

exit(0)

The startup and shutdown scripts must be executable in order for Ekos to invoke them (e.g. use chmod +x startup_script.py to mark the script as executable). Ekos Scheduler enables truly simple robotic operation without the need of any human intervention in any step of the process. Without human presence, it becomes increasingly critical to gracefully recover from failures in any stage of the observation run. Using KDE notifications, the user can configure audible alarms and email notifications for the various events in the scheduler.

Verifying Target Location

Due to a variety of factors, the captured target image might get shifted during the imaging process. Such factors might include:

  • Mount tracking issues.
  • Guiding issues.
  • External factors such as wind or vibrations.

To ensure the target stays centered, configure the scheduler to recheck the image position every few frames. If the captured image position has veered off the original target more than a configure threshold, then the scheduler would immediately abort the imaging and guiding processes, and perform a re-alignment to bring the mount back to the original target. Once this is successfully completed, it would resume the guiding and capturing workflows. You can configure these settings in Settings -> Configure KStars -> Ekos

recheck image

Manual Target Selection

When selecting a target usign the Ekos Find Tool, the objects equatorial coordinates (J2000) are automatically filled in the RA/DE fields. You can also define your own custom target. Simply type in your target name in the target field and then manually enter the desired J2000 coordinates in the respective fields.

KStars provides a conventient method to Copy Coordinates. Simply right-click on the desired object or point in the sky where you want to image, and click copy coordinates. You can paste the coordinates in any text editor and then copy back the J2000 coordinates in the scheduler RA and DE fields as shown below.

copy coordinates

Mosaic Planner

mosaic planner

Hubble-like super wide field images of galaxies and nebulae are truly awe inspiring, and while it takes great skills to obtain such images and process them; many notable names in the field of astrophotography employ gear that is not vastly different from yours or mine. I emphasize vastly because some do indeed have impressive equipment and dedicated observatories worth tens of the thousands of dollars. Nevertheless, many amateurs can obtain stellar wide-field images by combining smaller images into a single grand mosaic.

We are often limited by our camera+telescope Field of View (FOV). By increasing FOV by means of a focal reducer or a shorter tube, we gain a larger sky coverage at the expense of spatial resolution. At the same time, many attractive wide-field targets span multiple FOVs across the sky. Without any changes to your astrophotography gear, it is possible to create a super mosaic image stitched together from several smaller images. There are two major steps to accomplish a super mosaic image:

  1. Capture multiple images spanning the target with some overlap between images. The overlap is necessary to enable the processing software from aligning and joining the sub-images.
  2. Process the images and stitch them into a super mosaic image.

The 2nd step is handled by image processing applications such as PixInsight, among others, and will not be the topic of discussion here. The first step can be accomplished in Ekos Scheduler where it creates a mosaic suitable for your equipment and in accordance to the desired field of view. Not only Ekos creates the mosaic panels for your target, but it also constructs the corresponding observatory jobs required to capture all the images. This greatly facilitates the logistics of capturing many images with different filters and calibration frames across a wide area of the sky.

The Mosaic Planner in the Ekos Scheduler will create multiple Scheduler jobs based on a central target. To toggle the planner, click on the Mosaic Planner button in Ekos Scheduler or KStars INDI toolbar as illustrated in the screenshot. The planner draws the Mosaic Panel directly unto the sky map. It is recommended to enable HiPS overlay for the best experience. The planner is composed of four stages:

  1. Confirm Equipment: Ekos attempts to load equipment settings from INDI. If unsuccessful, you need to enter your equipment settings including your telescope focal length in addition to camera's width, height, and pixel dimensions. The settings are saved for future sessions.

confirm grip

  1. Select Grid: Select the mosaic panel dimension and overlap percentage. The Mosaic Panel is updated accordingly on the sky map. Adjust the Position Angle to match the desired mosaic orientation in the sky. If the Position Angle is different from your camera's usual orientation, you may need to rotate the camera either manually or via a mechanized rotator when the scheduler jobs are executed. Tile transparency is automatically calculated by default but may be turned off and adjusted manually. To compute the mosaic field from the number of tiles, click the Cover FOV button. The mosaic panel can be centered in the sky map by clicking on the Recenter button.

select grid

A large overlap will make frame stitching easier during post-processing, but it requires more panes to cover the desired extent. However, if you already know the minimal amount of sub-frames your rejection algorithm will use during post-processing, you may want to increase the overlap to attain that amount on the areas covered by multiple panes. For instance, a 4x4 mosaic grid with 75% overlap has 16 sub-frames covering the central intersection, which is enough for Winsorized Sigma rejection. Although the resulting stack does not have the same height on all parts of the final frame, this method gives you control on signal-to-noise ratio and allows you to provide context to your target while exposing a relatively low number of captures.

The large number drawn in the corner of each grid pane represents the order in which panes will be captured. The default S-shaped choice (west-east then alternating high-low/low-high moves), ensures minimal movement of the mount during observation. Uncheck Minimum mount move to revert to west-east/high-low movement only. The coordinates of each pane are rendered in their center as degrees, minutes and seconds. Finally, the angle each pane rotates from the center of the mosaic is displayed at the bottom. If your field of view is large, or if your mosaic is located close to a celestial pole, you may observe that rendered panes start rotating visibly due their horizontal position or high declination. Use the overlap to ensure panes cover the desired frame extents properly.

 mosaic grid
 
  1. Adjust Grid: Adjust Grid center by manually entering the J2000 center or by dragging the center of the mosaic on the sky map.

adjust grid

  1. Create Jobs: The final step is to select the sequence file and directory to store the images. Target field is automatically filled but may be changed as desired. Select the steps each scheduler job should execute in sequence (Track -> Focus -> Align -> Guide -> Capture), and adjust the frequency of automatic alignment and focus routines that must be executed during the mosaic operation. For example, if Align Every is set to 2 Scheduler Jobs, then the first job will run the astrometry alignment, while the second job will skip it. When the third job is executed, alignment is performed again and so forth

create scheduler jobs

Click Create Jobs to generate mosaic scheduler jobs and add them to the schedule queue.