Software - Revision history

Jv0059 at 08:31, 23 June 2023

2023-06-23T08:31:52Z

← Older revision		Revision as of 10:31, 23 June 2023
Line 38:		Line 38:
	*PyTorch [https://pytorch.org/]		*PyTorch [https://pytorch.org/]
	*TensorFlow [https://www.tensorflow.org/]		*TensorFlow [https://www.tensorflow.org/]
			*multiprocessing [https://docs.python.org/3/library/multiprocessing.html]
	'''Data visualization''':		'''Data visualization''':
	*seaborn [https://seaborn.pydata.org/]		*seaborn [https://seaborn.pydata.org/]
Line 68:		Line 69:
	* https://hclwizard.org/		* https://hclwizard.org/
	* http://hclwizard.org:3000/hclwizard/		* http://hclwizard.org:3000/hclwizard/

			== HPC (High-performance computing) commands ==
			Usefull commands based on slurm [https://slurm.schedmd.com/], which is mainly used on large clusters are:
			* sbatch [https://slurm.schedmd.com/sbatch.html]
			:- Useful is the --array option to submit a job array, using the index as an argument. This can be used, for example, to apply an analysis to many files or a large dataset. (e.g. sbatch --array=0-99 script.sh)
			* squeue -l
			* salloc
			* sinfo

Gk5026 at 13:20, 26 May 2023

2023-05-26T13:20:56Z

← Older revision		Revision as of 15:20, 26 May 2023
Line 61:		Line 61:


	== ~~'''~~ Colourblind Friendly Plotting Tools~~'''~~ ==		== Colourblind Friendly Plotting Tools==

	Here's a list of a few websites that have information and also palette generation for colourblind friendly plots:		Here's a list of a few websites that have information and also palette generation for colourblind friendly plots:

Gk5026 at 13:20, 26 May 2023

2023-05-26T13:20:40Z

← Older revision		Revision as of 15:20, 26 May 2023
Line 59:		Line 59:
	* Cartopy (python package) for maps [https://scitools.org.uk/cartopy/docs/latest/]		* Cartopy (python package) for maps [https://scitools.org.uk/cartopy/docs/latest/]
	* Matplotlib (standard python package) [https://matplotlib.org/]		* Matplotlib (standard python package) [https://matplotlib.org/]


			== ''' Colourblind Friendly Plotting Tools''' ==

			Here's a list of a few websites that have information and also palette generation for colourblind friendly plots:

			* https://colorbrewer2.org/#type=sequential&scheme=BuPu&n=9
			* https://hclwizard.org/
			* http://hclwizard.org:3000/hclwizard/

Gk5026 at 08:55, 26 May 2023

2023-05-26T08:55:52Z

← Older revision		Revision as of 10:55, 26 May 2023
Line 29:		Line 29:
	'''Data analysis''':		'''Data analysis''':
	*NumPy [https://numpy.org/]		*NumPy [https://numpy.org/]
	*xarray [https://docs.xarray.dev/en/stable/#]		*xarray [https://docs.xarray.dev/en/stable/#], easy handling of large nc files.
	*pandas [https://pandas.pydata.org/]		*pandas [https://pandas.pydata.org/]
	*SciPy [https://scipy.org/]		*SciPy [https://scipy.org/]
Line 58:		Line 58:
	* Paraview for 3D rendering [https://www.paraview.org/], note this one requires more computational power and should ideally be run through a supercomputer via VNC client.		* Paraview for 3D rendering [https://www.paraview.org/], note this one requires more computational power and should ideally be run through a supercomputer via VNC client.
	* Cartopy (python package) for maps [https://scitools.org.uk/cartopy/docs/latest/]		* Cartopy (python package) for maps [https://scitools.org.uk/cartopy/docs/latest/]
			* Matplotlib (standard python package) [https://matplotlib.org/]

Gk5026: Created page with " == Essential Software == The following is a list of essential software packages and the links to their web pages: Many of these can be installed using anaconda [https://www...."

2023-05-26T08:54:17Z

Created page with " == Essential Software == The following is a list of essential software packages and the links to their web pages: Many of these can be installed using anaconda [https://www...."

New page

== Essential Software ==
The following is a list of essential software packages and the links to their web pages:

Many of these can be installed using anaconda [https://www.anaconda.com/].

==== CDO ====
https://code.mpimet.mpg.de/projects/cdo/.

Note that CDO has a lot of built-in functions that are not well documented, but details about these can be usually found in their discussion forums, https://code.mpimet.mpg.de/projects/cdo/boards.

==== Python ====
Instructions on how to download and install Python for all OSs can be found at: https://www.python.org/.

It is usually recommended to use the Anaconda distribution to install Python. Details on how to do this are here: https://www.anaconda.com/

Python can be combined with a good Integrated Development Environment (IDE) of your choice. All existing IDEs have their pros and cons. Some of the most popular IDEs are the following:

*IDLE [https://docs.python.org/3/library/idle.html]
*PyCharm [https://www.jetbrains.com/pycharm/]
*Visual Studio Code[https://code.visualstudio.com/]
*Sublime Text [https://www.sublimetext.com/3]
*Jupyter [https://jupyter.org/]
*Atom [https://www.javatpoint.com/atom-python]
*Spyder [https://www.spyder-ide.org/]

Python boasts a large number of packages. Some of the most used packages for manipulating large files in NetCDF, HDF5 or CSV formats are the following:

'''Data analysis''':
*NumPy [https://numpy.org/]
*xarray [https://docs.xarray.dev/en/stable/#]
*pandas [https://pandas.pydata.org/]
*SciPy [https://scipy.org/]
'''Parallel computing, Machine learning, Deep learning etc.''':
*dask [https://www.dask.org/]
*scikit-learn [https://scikit-learn.org/stable/]
*Keras [https://keras.io/]
*PyTorch [https://pytorch.org/]
*TensorFlow [https://www.tensorflow.org/]
'''Data visualization''':
*seaborn [https://seaborn.pydata.org/]
*matplotlib [https://matplotlib.org/]
*Bokeh [https://docs.bokeh.org/en/latest/]
*plotly [https://plotly.com/]
*Geoplotlib [https://github.com/andrea-cuttone/geoplotlib]
Most of these packages are distributed through either '''Conda''' [https://docs.conda.io/projects/conda/en/latest/user-guide/concepts/installing-with-conda.html] or '''Pip''' [https://packaging.python.org/en/latest/tutorials/installing-packages/].

On top of these, various users also create packages tuned for specific purposes. They are usually made public through '''GitHub''' [https://github.com/python].

Various forums exist for the sole purpose of clearing specific questions about coding. One such forum is '''stackoverflow'''[https://stackoverflow.com/]. '''Medium''' [https://medium.com/] is also a good source for reading up about new ideas and tools in Python and other languages.

== Visualisation ==

The visualisation of "nc" files can be made easier using:

* NcView [https://cirrus.ucsd.edu/~pierce/software/ncview/quick_intro.html]
* Panoply [https://www.giss.nasa.gov/tools/panoply/]
* Paraview for 3D rendering [https://www.paraview.org/], note this one requires more computational power and should ideally be run through a supercomputer via VNC client.
* Cartopy (python package) for maps [https://scitools.org.uk/cartopy/docs/latest/]