Yes, indeed, we currently have limits in place. We're actively working to test our app and expand our infrastructure to handle various workloads before removing these limits.

For now, we're only converting SAS Files and Parquet files. Why? To be honest, we don't have a specific reason. We're currently focused on creating a Minimum Viable Product (MVP) for our project, and based on our team's experience, we've selected these formats.

Also, it's important to note that we limit the file size to 10MB. Additionally, files are retained for only 5 minutes after conversion for security reasons, regardless of whether they have been downloaded or not. Both the original and converted files will be deleted after this period.

Furthermore, please be aware that you can only download the converted file (in CSV format for now) once. If you miss the download button or have already downloaded the file, you will need to convert it again. This measure is in place to enhance security for unknown users. However, we plan to implement a dedicated area for registered users and revise our file retention policy accordingly.

We have created a discussion area in our Git repository for this purpose. If you'd like to get in touch with us, please click here to join the discussion.

A SAS file is a data file created by SAS (Statistical Analysis System), which is a software suite used for advanced analytics, business intelligence, and data management. SAS files typically have the extension ".sas7bdat" and contain structured data organized into rows and columns, similar to a spreadsheet or database table.

Here are some key points about SAS files:

• Format: SAS files are binary files specifically designed for storing and manipulating data within the SAS environment. They are not plain text files and require specialized software, such as SAS itself, to read and interpret the data.
• Data Types: SAS files can store various types of data, including numerical values, text strings, dates, and categorical variables. The data types and formats are defined within the SAS environment and can be customized based on the needs of the analysis.
• Metadata: Along with the actual data, SAS files also contain metadata that describes the structure of the dataset, including variable names, data types, formats, and any associated labels or attributes.
• Cross-Platform Compatibility: SAS files are platform-independent, meaning they can be created and accessed across different operating systems, including Windows, Linux, and Unix. This makes them suitable for data exchange and collaboration in multi-platform environments.
• Usage: SAS files are commonly used in data analysis, statistical modeling, and reporting tasks within the SAS environment. They serve as the primary data storage format for datasets generated or manipulated using SAS software.

Overall, SAS files play a crucial role in the SAS ecosystem, serving as the primary means of storing and working with structured data for analytical purposes.

Apache Parquet is an open-source, columnar data file format designed for efficient data storage and retrieval. It offers efficient data compression and encoding schemes to group complex data, improving performance. Apache Parquet is designed to be a common interchange format for batch and interactive workloads, similar to RCFile and ORC, other columnar storage file formats available in Hadoop.

Key Features:

• Columnar storage: Data is stored in columns, making it faster to access specific data points.
• Compression: Parquet files are compressed, saving storage space.
• Efficient encoding: Parquet uses efficient encoding schemes to further reduce file size.
• Open source: Parquet is an open-source format, meaning it is supported by a wide range of software.
• Flexible: Parquet can handle various data types, like numbers, text, and even Booleans (true/false values).

Benefits:

• Faster data retrieval: Due to the columnar structure, accessing specific data points within the file is faster compared to traditional row-based formats.
• Reduced storage needs: Compression helps save space on your hard drive.
• Wide software support: The open-source nature makes it compatible with many data processing tools.

Use cases:

• Data warehousing: Parquet is a good choice for data warehouses where data is stored in large tables and needs to be queried frequently.
• Big data analytics: Parquet is also a good choice for big data analytics where large datasets need to be processed quickly.
• Machine learning: Parquet uses efficient encoding schemes to further reduce file size.
• Open source: Parquet can be used to store training data for machine learning models.

Overall, Apache Parquet is a good choice for storing and processing large datasets. It offers efficient data compression and encoding, fast data retrieval, and wide software support.